Commit Graph
100 Commits
Author SHA1 Message Date
Xinyuan Tong afaa17a7f2 [Feature] Add --default-chat-template-kwargs server arg (#29579) 2026-07-13 11:34:39 -07:00
0663ebc783 [minimax-m3] Split 4/4: model + VL + glue + function-call + fp8 quant + generic infra (#28715)
Co-authored-by: Xinyuan Tong <xinyuan-tong@users.noreply.github.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-07-11 11:11:06 +08:00
Xinyuan Tong 6174f1cad8 [test] Set init-static attrs in mm_process_config mock fixtures (#30766) 2026-07-10 03:00:00 -07:00
Xinyuan Tong b76dd0be69 Fix Mistral GSM8K chat eval (#27757) 2026-07-09 21:08:48 -07:00
Xinyuan Tong 7132af28de Fix garbage output for bare-tekken Mistral checkpoints (e.g. Leanstral) (#30396) 2026-07-10 00:08:38 +05:30
Xinyuan Tong 074bb928f0 Move template manager files under parser; update CODEOWNERS (#26052) 2026-07-08 16:47:36 -07:00
Xinyuan Tong 937734d3ed ci(nightly): add force_baseline_update dispatch input for precision job (#30495) 2026-07-08 16:27:41 -07:00
Xinyuan TongandEazyReal 45019b56ce [Bugfix] Map reasoning_effort=low to Nemotron-3 Super low_effort + warn on unsupported levels (#30463)
Co-authored-by: EazyReal <8047065+EazyReal@users.noreply.github.com>
2026-07-08 12:19:44 -07:00
Xinyuan Tong 6f22790943 cookbook: add Hunyuan 3 (Hy3) Day-0 page (#30201) 2026-07-06 13:30:47 +08:00
Xinyuan Tong 854b46be99 feat(parser): resolve special-token suffix at runtime for compatibility (#29920) 2026-07-05 00:13:46 +08:00
Xinyuan Tong 9588cacaa1 Remove transformers 5.12.1 dead-code workarounds (#29758) 2026-07-03 00:03:06 +08:00
Xinyuan Tong 9ba4b8f8ba sgl-kernel: bump sgl-attn for varlen num_splits OOM fix (#29551) 2026-07-01 21:13:53 -07:00
Xinyuan TongandAlison Shao 0c1a0be3b2 fix(precision): do not promote failed runs to the comparison baseline (#28190)
Co-authored-by: Alison Shao <54658187+alisonshao@users.noreply.github.com>
2026-07-01 18:19:15 -07:00
Xinyuan Tong cf7c6ac234 fix(nightly-precision): pin flashinfer allreduce-fusion backend for TP-partial capture contract (#28925) 2026-07-01 17:26:19 -07:00
Xinyuan Tong 45314a9fcb [spec] Fix index_share_for_mtp_iteration being a no-op in EAGLE MTP draft (#29654) 2026-06-29 14:52:23 -07:00
Xinyuan Tong 38d4ffcd86 [cookbook] drop redundant serve flags (GLM-5.2) + fix M3 page-size note (#28731) 2026-06-29 13:34:13 +08:00
Xinyuan Tong ddc389cf09 [minimax-m3] Split 3/4: disagg K-only index-K transfer (#28714) 2026-06-28 13:54:42 +08:00
Xinyuan Tongandhzh0425 592f6c849b [minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool (#28713)
Co-authored-by: hzh0425 <hzh0425@apache.org>
2026-06-28 00:19:38 +08:00
Xinyuan Tong 30ea4c0f4b build(sgl-kernel): bump FlashMLA pin + fix cccl include for CUDA 13 (#29067) 2026-06-25 20:00:15 -07:00
Xinyuan Tong ed71fb8f95 fix(anthropic): detect-and-passthrough mid-conversation system messages (#28906) 2026-06-25 17:14:12 -07:00
Xinyuan Tong 0c6e8e9477 Expand parser auto detection coverage (#28449) 2026-06-23 12:26:37 -07:00
Xinyuan Tong de3ec2c437 [server_args] compute mem_fraction_static after dp chunked-prefill division (#28884) 2026-06-22 17:59:05 -07:00
6c212a5d6b [server_args] fix FA4 page_size auto-force for combined --attention-backend fa4 (#28825)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
2026-06-22 15:35:10 -07:00
Xinyuan Tong 7c23d2255a [minimax-m3] Split 1/4: sparse attention ops + JIT kernels + config foundation (#28712) 2026-06-22 13:10:43 -07:00
Xinyuan Tong db12bfcdc8 [JIT] Add kpool_topk_transform JIT kernel (#28670) 2026-06-22 01:04:21 -07:00
Xinyuan TongandXinyuan Tong 441ae9a5ae [Lint] Fix black formatting of DeepSeek-R1-MXFP4 MI35x tests (#28885)
Co-authored-by: Xinyuan Tong <justintong0323@gmail.com>
2026-06-22 14:20:34 +08:00
018d0c21dc [Docs] Add Anthropic-compatible API documentation (#28522)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 04:01:09 +00:00
Xinyuan Tong 61a8b42c00 docs(minimax-m3): add MMMU-Pro accuracy to B200 benchmark card (#28668) 2026-06-18 11:40:56 -07:00
Xinyuan TongandZijie Xia 72ccfec594 docs(cookbook): verify GLM-5.2 single-node B300 (FP8 + BF16) (#28460)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-06-17 03:47:32 +00:00
Xinyuan Tong 33f205d8c5 docs(cookbook): fix GLM-5.2 thinking toggle kwarg + document reasoning effort (#28454) 2026-06-16 18:17:34 +00:00
Xinyuan Tong 00081a00d5 docs(cookbook): tune GLM-5.2 MTP to 5-1-6 and simplify launch flags (#28448) 2026-06-17 01:18:34 +08:00
Xinyuan Tong 0cb6183432 docs(cookbook): add GLM-5.2 deployment cookbook (#28437) 2026-06-16 21:49:25 +08:00
Xinyuan Tongandzijiexia 33f99831f8 docs(minimax-m3): refresh B200 benchmarks (tp8, piecewise) + add GPQA (#28207)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-06-15 00:15:38 -07:00
Xinyuan Tong 1a66059c4e [Spec] Restore index_share_for_mtp_iteration in EAGLE V2 draft worker (#28192) 2026-06-14 18:01:11 -07:00
Xinyuan Tong 000fc975c7 ci(docker): support layered overlay images in release-docker-dev (#28206) 2026-06-14 16:34:02 -07:00
Xinyuan Tong 47fabb52ed docs(minimax-m3): add high-concurrency throughput tip for H200 bf16 (#28150) 2026-06-13 13:09:59 -07:00
85712fa5b0 Fix Responses API request handling (#25881)
Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org>
Co-authored-by: Kristin Cowalcijk <kristincowalcijk@gmail.com>
Co-authored-by: aerosta <63026763+aerosta@users.noreply.github.com>
Co-authored-by: glaziermag <glaziermag@users.noreply.github.com>
Co-authored-by: Blake Ledden <blake.ledden@gmail.com>
Co-authored-by: PanJason <pyyjason@gmail.com>
Co-authored-by: Leoyzen <leoyzen@gmail.com>
Co-authored-by: kennyu <966806+kennyu@users.noreply.github.com>
2026-06-12 14:47:55 -07:00
+3 b3270264e4 Fix Anthropic Messages API compatibility (#25876)
Co-authored-by: Jairo David Campaña Rosero <jairocampana10001@gmail.com>
Co-authored-by: Karan Bansal <3264937+karanb192@users.noreply.github.com>
Co-authored-by: eason <85663565+mango766@users.noreply.github.com>
Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: qingchanghan <17794466+qingchanghan@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Ajay Anubolu <124525760+AjAnubolu@users.noreply.github.com>
Co-authored-by: Ravitez Dondeti <13931987+dondetir@users.noreply.github.com>
Co-authored-by: Ratish P <114130421+Ratish1@users.noreply.github.com>
Co-authored-by: Xiaoshuai Zhang <15795935+jetd1@users.noreply.github.com>
Co-authored-by: Ricardo-M-L <69202550+Ricardo-M-L@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuan.tong@radixark.ai>
2026-06-12 14:46:57 -07:00
Xinyuan Tong 9f6b2339f9 docs(minimax-m3): warm-steady-state benchmark numbers (#28062) 2026-06-12 08:42:23 -07:00
Xinyuan Tong dba617f2ec doc: update docs for new model (#28060) 2026-06-12 21:14:35 +08:00
Xinyuan Tong 66ab5c9c7a fix(gateway): make sgl-model-gateway a cargo workspace so maturin 1.14 accepts the parent README (#27997) 2026-06-11 23:37:29 -07:00
Xinyuan Tong 7425bebb6c docs(cookbook): restore Gemma 4 transformers commit pin (#27321) 2026-06-04 17:43:36 -07:00
Xinyuan Tong 45a66f4088 [Docs] Update unified Text/Vision/Audio model cookbook: install + sgl-eval accuracy (#27171) 2026-06-03 09:32:08 -07:00
Xinyuan Tong fa5c8a3101 [model] support encoder-free unified Text/Vision/Audio model (#27167) 2026-06-03 23:58:06 +08:00
Xinyuan Tong 79c844527c Upgrade xgrammar to 0.2.1 (#25676) 2026-05-29 11:40:07 +08:00
Xinyuan Tong bed20249f1 fix(tool_call): reland schema type normalization (#26433) 2026-05-28 14:31:18 +08:00
Xinyuan Tong 64c7c6851b fix(tool_call): normalize non-standard JSON Schema types in tool params (#23476) 2026-05-26 15:23:59 +08:00
Xinyuan Tong 40faf44f7a [auto-detect] match Ring-2.6/Ling XML kv tool-call format via vocab signature (#25366) 2026-05-20 23:34:52 -07:00
Xinyuan TongandXinyuan Tong 52eebc82ae [Docs] MiMo-V2.5 cookbook: B200 benchmarks + multi-layer EAGLE acceptance profile + long-context reference (#25359)
Co-authored-by: Xinyuan Tong <xinyuan.tong@radixark.ai>
2026-05-19 23:15:21 -07:00
Xinyuan Tong 0aedc5678b loader: yield filtered MTP weights lazily to avoid OOM hang on multi-layer EAGLE (#25748) 2026-05-20 12:33:54 +08:00
Xinyuan Tong aad00b0ed8 Upgrade transformers to 5.8.1 (#25451) 2026-05-19 22:20:30 +08:00
Xinyuan Tong 7cb4669a04 [Fix] DeepSeek-V3.2: build structural tag locally to encode both wrapper and invoke layers (#25233) 2026-05-15 14:32:50 -07:00
Xinyuan Tong 33f1d3915f [NEW MODEL] Add H200 validation for Ring-2.6-1T cookbook (#25370) 2026-05-15 11:47:15 -07:00
Xinyuan Tong c3daa77e9a [NEW MODEL] Add Ring-2.6-1T cookbook (#25360) 2026-05-14 23:32:22 -07:00
Xinyuan Tong 5b589ed2e7 feat(constrained): two-phase reasoning grammar + --enable-strict-thinking (#23953) 2026-05-07 14:21:51 -07:00
Xinyuan Tong af2a2ac618 fix(function_call): handle Kimi-K2.5 bare numeric tool call IDs (#23950) 2026-05-07 14:20:02 -07:00
Xinyuan Tong d8f9d32a05 feat(reasoning): auto-detect reasoning/tool-call parser from chat template (#23952) 2026-05-07 14:19:16 -07:00
Xinyuan Tong f1395af543 fix(openai): map reasoning.enabled to thinking AND enable_thinking (#23951) 2026-05-07 14:01:35 -07:00
Xinyuan Tong 1e404afec2 fix(req_pool): bump pool.size to match actual tensor row count after #24243 (#24439) 2026-05-05 16:58:26 -07:00
Xinyuan Tong 8d1b6f0c00 Add zRzRzRzRzRzRzR to CI permissions (#24432) 2026-05-05 12:37:50 -07:00
Xinyuan Tong 989a16187d [Bench] Fix bench_serving missing reasoning_content stream chunks (#23954) 2026-04-30 15:00:27 -07:00
Xinyuan Tong 1376761841 fix(moe): repair dead import in fused_moe_native after MoE refactor (#24069) 2026-04-29 11:14:52 -07:00
Xinyuan Tong 4cf109bbd1 debug followup (#24058) 2026-04-29 23:03:27 +08:00
Xinyuan Tong 1279ae0787 Bugfix (#24027) 2026-04-29 21:13:51 +08:00
Xinyuan Tong 832b4f59ed [Bench] fix MMMU answer-extraction regex dropping multi-line responses (#23864) 2026-04-29 14:48:49 +08:00
Xinyuan Tong e458a9248f docs: enable MiMo V2.5 MTP cookbook path (#23945) 2026-04-28 10:22:19 -07:00
Xinyuan Tong 3fce8f2009 [Docs] add cookbook for Ling-2.6 family (#23947) 2026-04-29 00:42:04 +08:00
Xinyuan Tong f34222da1b [Docs] add cookbook for MiMo-V2.5 family (#23851) 2026-04-28 01:38:41 +08:00
Xinyuan Tong 96b0c64c88 Add docs_new code owner (#23855) 2026-04-27 10:31:35 -07:00
Xinyuan Tong e5198386bd Upgrade transformers from 5.5.4 to 5.6.0 (#23525) 2026-04-26 22:33:54 -07:00
6d03861476 support Hy3 preview (#23533)
Co-authored-by: pengmeng <pengmeng@tencent.com>
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
Co-authored-by: chengvjiang <chengvjiang@tencent.com>
Co-authored-by: russellfeng <russellfeng@tencent.com>
2026-04-24 12:03:24 -07:00
Xinyuan Tong 4868e367f8 docs: add Hunyuan 3 Preview cookbook (#23532) 2026-04-23 02:44:47 -07:00
Xinyuan Tong de962f3274 docs(cookbook): add Qwen3.6-27B dense variant (#23486) 2026-04-23 01:22:46 +08:00
f63def8510 [XPU] Fix DeepSeek-OCR tests under transformers 5.x (#23044)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-04-21 14:57:56 +08:00
Xinyuan Tong 082eaed0a4 test: fix flaky required function calling assertion (#22890) 2026-04-16 09:44:26 -07:00
Xinyuan Tong 34fef07a15 Upgrade transformers to 5.5.3 and refactor hf_transformers_utils into subpackage (#21569) 2026-04-15 20:03:44 -07:00
Xinyuan Tong 3f60df8012 [Test] Fix flaky test_function_call_specific by adding strict=True (#22624) 2026-04-12 01:22:57 -07:00
Xinyuan Tong 9a4e8089ff [Whisper] Batch encoder forward for concurrent prefill requests (#22361) 2026-04-12 14:15:14 +08:00
Xinyuan Tong 7c6db40540 Fix tool call constrained decoding and parsing for models with native formats (#21593) 2026-04-10 20:37:23 -07:00
2813cb6d9a [New Model] Gemma 4 (#21952)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Pengyu Chen <pychen96@gmail.com>
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Andy Luo <andy.luo@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: adarshxs <adarsh.shirawalmath@gmail.com>
2026-04-06 20:24:44 -07:00
Xinyuan Tong 87a2768269 VLM: change default mm-attention backend from triton_attn to fa4 (on blackwell) (#21595) 2026-04-01 14:29:59 +08:00
Xinyuan Tong ced69c9f84 feat: enable CUDA graph and timestamp for the whisper model(#21190) 2026-03-29 01:46:03 +08:00
Xinyuan Tong a02cff7f2b [Fix] Patch is_flash_attn_2_available for flash-attn-4 in VLM input format test (#20946) 2026-03-19 13:00:51 -07:00
6b8a6545b2 Add Mistral Small 4 (Pixtral) support (#20708)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Alex Nails <alexnails@radixark.ai>
Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
Co-authored-by: dbari <dbari@users.noreply.github.com>
2026-03-18 14:15:32 -07:00
d1e95af282 Upgrade transformers==5.3.0 (#17784)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Kangyan-Zhou <zky314343421@gmail.com>
Co-authored-by: Alison Shao <alisonshao@mac.lan>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-03-18 13:50:43 -07:00
Xinyuan Tong c330b687a1 [Bugfix] Fix GLM-4.6V vision regression in glm4v_moe and glm_ocr (#20463) 2026-03-13 21:48:28 -07:00
Xinyuan Tong 11d9c36c2f Replace soundfile+torchaudio with torchcodec AudioDecoder in load_audio (#20190) 2026-03-10 17:26:29 +00:00
Xinyuan Tong d116a8cd94 [Bugfix] Fix load_audio: mono before resample + use torchaudio (#20054) 2026-03-09 19:24:20 +08:00
4a757990a1 [VLM] Replace decord with torchcodec for video decoding (#20055)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: BakerBunker <17872844+BakerBunker@users.noreply.github.com>
2026-03-09 19:23:49 +08:00
Xinyuan Tong 9ebffef1ef [FIX] NSA backend page_table overflow in speculative decoding target_verify (#19016) 2026-03-05 16:04:58 -08:00
Xinyuan Tong dbf1247fe0 Add KimiK2Detector with tool interruption support (#19696)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-03-03 14:04:49 +08:00
581bf53e03 Whisper model support & /v1/audio/transcriptions endpoint & benchmark (#16983)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-23 17:28:37 -08:00
Xinyuan Tong 677b66af80 fix KimiK2Detector regex patterns with re.DOTALL (#19120)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-02-21 22:13:08 +08:00
Xinyuan Tong 4a362a0e04 fix tool handling in OpenAIServingChat (#18996)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-02-21 22:07:09 +08:00
Xinyuan Tong cc451671b5 [FEAT] Add Anthropic compatible API endpoint (#18630)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-02-21 19:37:38 +08:00
398b81f78c Support GlmMoeDsaForCausalLM (#18521)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Signed-off-by: BBuf <1182563586@qq.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-02-10 15:20:10 +08:00
e8a2c13380 Deepseekv32 compatibility with transformers v5 (#18297)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
2026-02-10 14:50:40 +08:00
Xinyuan Tong 0b4d4f2838 Fix MMLU benchmark to auto-download data and resolve path issue (#18486) 2026-02-09 10:40:40 -05:00
Xinyuan Tong 9409c43593 Fix flaky tool calls in the Kimi K2.5 model (#17914)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-01-28 20:58:16 -08:00
Xinyuan Tong 3c34d2c3eb [FIX] kimi_k2 reasoning parser (#17901)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-01-28 19:47:09 -08:00