minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
10,201 Commits 3 Branches 0 Tags
3f9fc8b848365a5797a44856854e3e6f00a60dd0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Xinyuan TongMahmoudAshraf97gemini-code-assist[bot]
 		581bf53e03 Whisper model support & /v1/audio/transcriptions endpoint & benchmark (#16983) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-02-23 17:28:37 -08:00
Xinyuan Tong 677b66af80 fix KimiK2Detector regex patterns with re.DOTALL (#19120) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-02-21 22:13:08 +08:00
Xinyuan Tong 4a362a0e04 fix tool handling in OpenAIServingChat (#18996) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-02-21 22:07:09 +08:00
Xinyuan Tong cc451671b5 [FEAT] Add Anthropic compatible API endpoint (#18630) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-02-21 19:37:38 +08:00
Xinyuan TongXiaoyu ZhangBBuf
 		398b81f78c Support GlmMoeDsaForCausalLM (#18521) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Signed-off-by: BBuf <1182563586@qq.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
 2026-02-10 15:20:10 +08:00
Xinyuan TongXiaoyu ZhangBaizhou Zhang
 		e8a2c13380 Deepseekv32 compatibility with transformers v5 (#18297) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2026-02-10 14:50:40 +08:00
Xinyuan Tong 0b4d4f2838 Fix MMLU benchmark to auto-download data and resolve path issue (#18486) 2026-02-09 10:40:40 -05:00
Xinyuan Tong 9409c43593 Fix flaky tool calls in the Kimi K2.5 model (#17914) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-01-28 20:58:16 -08:00
Xinyuan Tong 3c34d2c3eb [FIX] kimi_k2 reasoning parser (#17901) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-01-28 19:47:09 -08:00
Xinyuan Tong 37c04c2245 fix: Refactor register_image_processor to use kwarg instead of positional arg (#17685) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-01-24 15:31:01 -08:00
Xinyuan Tong 2069050d3f fix: Handle multiple named chat templates in HuggingFace tokenizers (#17236) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-01-18 17:20:04 +08:00
Xinyuan Tong ae434f7821 Move limit-mm-data-per-request to make code clean (#15775) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-12-24 10:59:39 -08:00
Xinyuan Tong 47cdb65a45 fix: update argument extraction in R1 chat template (#15547) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-12-21 09:18:49 +08:00
Xinyuan Tong 0a346d3bd9 feat: Add limit-mm-data-per-request argument to server arguments (#15418) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-12-20 10:44:47 -08:00
Xinyuan TongmomaekMuqi Li
 		41683536d3 feat: DeepSeek-V3.2 Streaming tool call output (#15278) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: momaek <momaek17@gmail.com>
Co-authored-by: Muqi Li <muqi1029@gmail.com>
 2025-12-17 17:43:58 -08:00
Xinyuan Tong 96705514bd fix: dpskv32 chat history processing, default drop_thinking to true (#15064) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-12-13 12:16:04 -08:00
Xinyuan Tong 8bf10e71cd remove dpsk3.2 sys prompt (#14923) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-12-11 20:26:13 -08:00
Xinyuan Tong 4885f8b9c1 fix: handle Jinja2 template errors as client errors in OpenAIServingChat (#14748) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-12-11 17:06:26 -08:00
Xinyuan Tong 9975acf50f [refactor] Update reasoning parameter to require_reasoning (#14922) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-12-11 16:55:59 -08:00
Xinyuan Tong fd1ebbb0d6 update mistral detector (#14921) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-12-11 16:43:36 -08:00
Xinyuan TongandYueming Yuan 6d37e70883 ministral3 (#14251) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Yueming Yuan <yy28@illinois.edu>
 2025-12-04 14:31:26 -08:00
Xinyuan Tong a355794905 Expend compatibility check for all quantized MoE models (#13465) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-11-20 01:24:27 +08:00
Xinyuan TongalisonshaoMickBaizhou ZhangKangyan-Zhou
 		b1c688fba2 refactor: cleanup vision attention related codes (#13228) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: alisonshao <54658187+alisonshao@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: Kangyan-Zhou <zky314343421@gmail.com>
 2025-11-16 22:44:40 +08:00
Xinyuan Tong 4983b7e7aa Fix strict level setting for Kimi K2 tool calls when not explicitly set (#13077) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-11-12 00:01:42 -08:00
Xinyuan Tong d2a8f71c2f [feat] Add SGLANG_TOOL_STRICT_LEVEL for tool-call behavior control (#12423) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-11-01 13:15:02 -07:00
Xinyuan Tong 0bfe1d145c fa3 & trtllm_mha spec overlap (#11874) 2025-10-31 17:38:13 -07:00
Xinyuan Tongandxinyuant 684864814b Feat: deepseek-ocr logits processor (#12415) 
Co-authored-by: xinyuant <xinyuant@usc.edu>
 2025-10-31 23:35:22 +08:00
Xinyuan TongandCursor Agent 82cfcd3bb8 [Refactor] tuning_fused_moe for MLLM and small refactor (#11224) 
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
 2025-10-31 08:54:14 +08:00
Xinyuan Tong 729f612dc6 Update openai package version to 2.6.1 (#12222) 2025-10-28 11:23:40 +08:00
Xinyuan Tong 3f4cc0aff0 Remove description for --enable-beta-spec argument (#12177) 2025-10-26 16:49:39 -07:00
Xinyuan Tong b9fb74f3bc fix: bench_serving ITL calculation when using spec-decoding (#12064) 2025-10-24 17:02:44 +08:00
Xinyuan Tong 1f106ee365 [grammar] Avoid server crash when grammar backend is None (#11401) 2025-10-09 18:38:10 -07:00
Xinyuan Tong c4d77774e1 update sampling_params documentation with defaults (#11315) 2025-10-07 18:36:26 -07:00
Xinyuan Tong e3c7f09146 Update tool parser and related documentation (#11223) 2025-10-07 11:03:40 -07:00
Xinyuan Tong fd8a0b29c0 fix: correct scale parameter remapping logic in Llama4ForConditionalGeneration (#11282) 2025-10-06 17:28:23 -07:00
Xinyuan Tong 0cd1996eae feat: add shortcut detection for multimodal templates in Jinja format (#11209) 2025-10-06 04:13:17 +00:00
Xinyuan Tongandyhyang201 652c24a653 Update transformers package version to 4.57.0 (#11222) 
Co-authored-by: yhyang201 <yhyang201@gmail.com>
 2025-10-05 23:45:14 +00:00
Xinyuan TongandMick a9ce2bcb3c [Doc] Update multimodal language models documentation (#11111) 
Co-authored-by: Mick <mickjagger19@icloud.com>
 2025-09-30 22:10:31 -07:00
Xinyuan Tong 1193f13181 fix: KimiK2Detector Improve tool call ID parsing with regex (#10972) 2025-09-30 17:44:44 -07:00
Xinyuan Tongyhyang201Yangmin LiYuan Luogemini-code-assist[bot]
 		62e2e99db6 fix: make inference deterministic for large TP (#10930) 
Co-authored-by: yhyang201 <yhyang201@gmail.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: Yuan Luo <yuan.luo@hotmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-09-27 02:46:45 -07:00
Xinyuan Tong 6088548216 Update CODEOWNERS to include JustinTong0323 in FC (#10939) 2025-09-25 22:55:56 -07:00
Xinyuan Tong 71f24ef8f6 feat: add cache_salt support to request (#10718) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-09-23 23:30:25 -07:00
Xinyuan TongandQiaolin Yu aab35bccb4 fix: draft model IMA by overide max_positional_embeddings (#10787) 
Co-authored-by: Qiaolin Yu <qy254@cornell.edu>
 2025-09-23 12:56:16 -07:00
Xinyuan Tong 12d6cf18f0 Refactors radix cache for extra key support (#10317) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-09-22 02:16:16 +08:00
Xinyuan Tongandgemini-code-assist[bot] 6f4676ef85 fix: tool parse in large streaming chunk beginning with normal content (#10397) 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-09-12 22:29:35 -07:00
Xinyuan Tong f3b5db6ee8 Feat: support disable tool parser (#10184) 2025-09-10 14:03:55 -07:00
Xinyuan Tong 8116804e4f Fix: (glm4v) Add missing field (#10147) 2025-09-07 21:47:14 -07:00
Xinyuan TongandBinyao Jiang f3440adcb5 vlm: enable GLM4.1V server testing & fix video processing (#10095) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Binyao Jiang <byjiang1996@gmail.com>
 2025-09-08 03:53:08 +01:00
Xinyuan Tong 273b28344b [Minor] Refactors KV memory pool (#9842) 2025-09-05 17:06:08 -07:00
Xinyuan Tong 56eb5d0a3d fix swa clear(): rename is_in_free_group to is_not_in_free_group (#9914) 2025-09-03 11:42:12 -07:00
Xinyuan Tong f84b57c80e Move git clone command up from README (#9740) 2025-08-28 00:27:00 -07:00
Xinyuan Tong 68a54e063e Sets default model name in request classes (#9683) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-08-27 10:43:03 -07:00
Xinyuan Tong ca4b86c564 fix: Update OpenAI client base URL in documentation (#9576) 2025-08-24 23:06:57 -07:00
Xinyuan Tong fedfe91c1a [Docs] Add doc and quick demo for gpt-oss responses api & buildin tools (#9497) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-08-21 23:51:52 -07:00
Xinyuan Tong 13ec8d427e [Docs]Update reasoning parser doc & fix outdated link (#9492) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-08-21 22:08:28 -07:00
Xinyuan Tong 0b3a5b1151 Update reasoning parser doc (#9468) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-08-21 17:25:30 -07:00
Xinyuan Tong 6c855db82c Revert "bugfix: Fix output_ids extraction in detokenizer_manager" (#9467) 2025-08-21 17:24:25 -07:00
Xinyuan Tong e8449ab515 Add deepseek v3.1 thinking parser support and update docs (#9464) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-08-21 15:09:40 -07:00
Xinyuan Tong 84719b527a fix: InternS1 don't recognize image, updates image token for InternVL processor (#9381) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-08-20 16:43:03 -07:00
Xinyuan Tong 3e7ff1ab1f fix: reasoning parser when request have enable_thinking flag (#8933) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-08-07 15:52:06 -07:00
Xinyuan Tong c36a6693f3 Disable gemma3 for SWA due to CUDA illegal memory access error (#8895) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-08-07 00:44:44 -07:00
Xinyuan TongandChang Su 3fa3c6cd6a Enables force reasoning based on chat template for Qwen3-Thinking (#8369) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
Co-authored-by: Chang Su <csu272@usc.edu>
 2025-08-06 20:02:47 -07:00
Xinyuan Tong 7e831efee8 Fix chat template handling for OpenAI serving (#8635) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-07-31 21:49:45 -07:00
Xinyuan Tong b602f42354 Urgent Fix: intern-s1 chat-template matching (#8403) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-27 00:22:31 -07:00
Xinyuan TongandXinyuan Tong 38000a5f44 Fix gemma3n with hybrid swa (#8240) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-07-23 13:29:18 -07:00
Xinyuan TongandXinyuan Tong 70251e935e fix: match chat-template for internvl3 (#8262) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-07-23 13:29:03 -07:00
Xinyuan TongandXinyuan Tong 69adc4f81c fix: retrieve mm token by modality, raise error if none (#8221) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-07-22 08:06:35 +08:00
Xinyuan Tong 8430bfe3e9 [Refactor] simplify multimodal data processing (#8107) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-20 21:43:09 -07:00
Xinyuan Tong 7498522f7d update transformers to 4.53.2 (#8029) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-15 18:24:39 -07:00
Xinyuan Tong 9120e83d03 fix: remove redundant rotary embedding cache recomputation in MiniCPM (#8022) 2025-07-15 00:12:45 -07:00
Xinyuan Tong 6e923dbd30 feat: update multimodal data handling in engine entrypoint (#8002) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-15 00:12:22 -07:00
Xinyuan Tong e5888eddda Fixes typo in assertion message (#7895) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-09 01:47:14 -07:00
Xinyuan Tong 136c6e0431 fix: Handles input_embeds in GenerateReqInput when n>1 (#7830) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-08 14:00:42 -07:00
Xinyuan Tong 43e20c0647 Support Mimo-VL (#7579) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-08 14:00:25 -07:00
Xinyuan Tong 4bab50a6b5 Fix llama4 vision (#7840) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-08 14:00:03 -07:00
Xinyuan Tong 43f93f632c fix CI: update native api ipynb (#7754) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-07-03 15:25:00 -07:00
Xinyuan Tong 3a911b854d Refactor mm processors and Enable mixed modality processing (#7629) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-06-30 23:14:48 -07:00
Xinyuan Tong 8f335b5bd6 Fix stream reasoning parser and Adds Kimi reasoning parser (#7432) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-06-29 14:39:05 -07:00
Xinyuan Tongyhyang201Mick
 		c45e49d817 oai: Adds support for OpenAI chat completions API in bench_serving (#7036) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
Co-authored-by: yhyang201 <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
 2025-06-28 22:59:20 +00:00
Xinyuan Tong 357921aa51 Fix: Minicpm (#7612) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-06-27 17:32:29 -07:00
Xinyuan Tong 1b95162008 Updates transformers and timm dependencies (#7577) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-06-27 00:30:17 -07:00
Xinyuan Tong 604efe07e1 Updates Gemma3n MLP layer to adapt latest transformers version (#7573) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-06-26 15:07:22 -07:00
Xinyuan Tongyhyang201Mick
 		9b00990bea chore: remove vlm unnecessary import (#7541) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
Co-authored-by: yhyang201 <yhyang201@gmail.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
 2025-06-26 01:38:15 -07:00
Xinyuan TongandChang Su 0998808009 Refine OpenAI serving entrypoint to remove batch requests (#7372) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
Co-authored-by: Chang Su <csu272@usc.edu>
 2025-06-20 14:33:43 -07:00
Xinyuan Tong 70c471a868 [Refactor] OAI Server components (#7167) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-06-16 20:45:20 -07:00
Xinyuan Tong 697b0f71f0 [Refactor] image data process in bench_serving (#6879) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-06-06 21:11:17 -07:00
Xinyuan Tong cf9815ba69 [Refactor] Multimodal data processing for VLM (#6659) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-06-04 11:22:33 -07:00
Xinyuan Tong d6864ce6d6 [New Model] Devstral support (#6547) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-05-26 19:27:48 -07:00
Xinyuan Tong 681fdc264b Refactor vlm embedding routine to use precomputed feature (#6543) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-05-24 18:39:21 -07:00
Xinyuan Tong 16d4f6801b doc: Update README.md with adding deepwiki badge to enable weekly auto-refresh (#6508) 2025-05-21 16:27:34 -07:00
XinyuanTong 9d8ec2e67e Fix and Clean up chat-template requirement for VLM (#6114) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-05-11 00:14:09 +08:00
XinyuanTong e88dd482ed [CI]Add performance CI for VLM (#6038) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-05-07 19:20:03 -07:00
XinyuanTong 6ea1e6ac6e Support MMMU benchmark for InternVL (#5968) 2025-05-02 00:17:21 -07:00
XinyuanTong c5645e928f feat: add concurrency evaluation logic in mmmu benchmark (#5782) 2025-05-01 18:20:08 -07:00
XinyuanTong e97e57e699 Remove unused method calculate_num_image_tokens from qwen2_vl.py (#5783) 2025-04-30 17:46:59 -07:00
XinyuanTong 8465f035d1 Add qwen3 30b fused moe config (#5859) 2025-04-29 00:24:00 -07:00
XinyuanTong 0045f4b2af feat: Add fused moe triton config for qwen3 moe on h100 (#5833) 2025-04-28 08:37:13 -07:00
XinyuanTong e7beff8a13 fix: examples for token_in_token_out_vlm (#5193) 2025-04-11 01:38:23 -07:00
XinyuanTong d09a51f1f6 [feat&refactor] Enhance multimodal input support with refactor io_struct (#4938) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
 2025-04-08 14:48:07 -07:00
XinyuanTongandGeLee 9eb49e878b [VLM RLHF] Take Image input for verl vlm rollout (#4915) 
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
Co-authored-by: GeLee <leege233@gmail.com>
 2025-04-01 20:03:17 -07:00