Commit Graph
79 Commits
Author SHA1 Message Date
Colin Z 875dd41e6f [AMD] Fix Load and Inference of MLA models with Quark PTPC FP8 attention on ROCm (#28734) 2026-09-29 11:20:10 -07:00
Xiaoyu Zhang 38ec649048 [Refactor] Restore logical kernel groups and test organization (#41243) 2026-09-26 12:30:44 +08:00
Liangsheng Yin f35ec18d8b [Test] Remove more unit tests that mirror implementation or never run in CI (#41297) 2026-09-25 19:22:14 -07:00
f4d9d2e670 [RL] Add RL weight-update sessions and support updating spec draft runners (#40777)
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Nan Jiang <59716405+nanjiangwill@users.noreply.github.com>
Co-authored-by: maocheng23 <maocheng@berkeley.edu>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: JD-ETH <jaedon.guo@gmail.com>
Co-authored-by: Letian Ruan <ruanlt420@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-23 20:04:57 -07:00
Liangsheng Yin 78980a3b0b [misc] Remove deprecated endpoints, env vars and aliases past two releases (#40795) 2026-09-22 18:14:45 -07:00
c19dc43cc7 [AMD] [GLM-5.3-Flash Day 0] Load the MXFP4 MTP draft layer (#39779)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: CHANG <jacchang@crsuse2-slog-003.us-east2-a.compute.internal>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
2026-09-22 15:07:51 -07:00
YAMY 9b59fc5db5 [ModelOpt][PP] Keep BF16 shared experts out of the NVFP4 fusion so TP1 pipeline stages can load (#40628) 2026-09-21 21:45:58 -07:00
e1daf68304 [AMD] [GLM-5.3-Flash Day 0] Honor fused and per-expert names in quark exclude (#39317)
Co-authored-by: Yikai Zhang <ykzhang12@gmail.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 21:26:42 -07:00
b44e248682 [AMD] [GLM-5.3-Flash Day 0] Enable FP8 and Quark MXFP4 MoE on gfx950 (#38546)
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 21:07:37 -07:00
Cheng Wan acac4dd9d9 [Refactor] Clean up parallel runtime comments (#40632) 2026-09-21 14:32:22 -07:00
Liangsheng Yin 76a9065bef [Fix] Raise on undelivered embeddings in send_with_url, fix broken tests (#40502) 2026-09-20 17:35:27 -07:00
42875bcd2a fix(modelopt): dispatch NVFP4 MoE on the cached backend, not the live global (#38932)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-09-20 19:28:54 -04:00
DarkSharpnessandBBuf d1acbe0746 [DSV4.1] Big fused wo_a quant (#39957)
Co-authored-by: BBuf <1182563586@qq.com>
2026-09-19 19:46:53 +08:00
Cheng Wan 567d5925fe Fix mxfp4 padding test stubbing an accessor the module no longer imports (#40308) 2026-09-19 00:43:56 -07:00
Liangsheng Yin d507accadc [Test] Drop dead and strictly-subsumed CI test registrations (#40264) 2026-09-18 17:49:13 -07:00
Cheng Wan afe71f4b9e Read process groups through the runtime context (#40068) 2026-09-18 17:40:32 -07:00
Sam (Kesen Li) d346b214fb feat(kv-cache): support SM100 NVFP4 GenMHA and speculative decoding (#36340) 2026-09-18 14:50:46 -07:00
a6cf05817f dsv4.1: remaining model and runtime integration (#38798)
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <xiaoyu.zhang@radixark.ai>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Zhichen Zeng <zczeng@uw.edu>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
2026-09-18 02:55:30 -07:00
Liangsheng Yin 1b200ffaaa [Quant] Serve 32-wide-K ue8m0 block-FP8 linears through the FlashInfer MXFP8 GEMMs (#40039) 2026-09-18 02:51:12 -07:00
cebca698e2 [Qwen3.8] Enable NVIDIA NVFP4 on DGX Spark with file-backed PLE and PDL router fix (#39126)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: rdxa <rdxa@rdxa-int-spark-01.yvb.moe>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: Manrique <nanomlm@gmail.com>
Co-authored-by: yhyang201 <yhyang201@gmail.com>
2026-09-13 16:23:41 +08:00
Tuan Nguyen Gia 7c195b9151 [AMD] Fix Quark load of MiniMax-M3 MXFP4 index_qkv_proj (#37254) 2026-09-12 00:50:41 -07:00
Brayden ZhongandBrayden Zhong c0b790cf7f Delete cutlass_mla, non-Marlin GPTQ, AWQ AOT kernel, and Dual Chunk Flash Attention (#32114)
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
2026-09-10 15:12:01 +08:00
Mohammad Miadh AngkadandMohammad Angkad a711785475 [Kernel] Drop the vendored dense BF16 GEMM port in favor of FlashInfer 0.6.18 (#38124)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-07 12:59:33 -07:00
214313ee79 Fuse Nemotron latent MoE projection and shared add (#30430)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
2026-09-07 10:26:19 +08:00
sglang-botandsglang-bot 6252993afe chore: update CI test est_time values (#38238)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-09-06 17:49:41 -07:00
Beihao Zhou 1e6f18bfeb [MoE Refactor] Migrate SM100 trtllm-gen mxfp4 MoE onto MoeRunner (#32405) 2026-09-05 13:48:10 +00:00
kkandwunhuang a6001478f4 [AMD] Perf Kimi-K3 MoE optimization (#33838)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-09-03 02:28:28 -07:00
Alex NailsandAlison Shao 28262c20df [CI][RFC] Replace black-jupyter with ruff-format (#37210)
Co-authored-by: Alison Shao <a.shao@wustl.edu>
2026-09-02 19:46:08 -07:00
Lee NauandYangmin Li 6e41f1ad29 [Fix] Preserve FP32 in SM107 MXFP8 fallback (#37489)
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
2026-09-02 17:55:14 -07:00
26f760d5c0 [CPU] Support FP8 KV cache (#32733)
Co-authored-by: Zheng, Beilei <beilei.zheng@intel.com>
Co-authored-by: mingfeima <mingfei.ma@intel.com>
2026-09-02 10:20:54 +08:00
Yuan Luoandluoyuan.luo 5b04408784 [MoE] Add FlashInfer SM90 MXFP4 W4A8 CUTLASS MoE (#34967)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-08-31 20:04:41 -07:00
Cheng Wan 7e751153eb [Config] Round 5.1: the published-side readers ask the bags, and a platform fact gets one address (#37086) 2026-08-30 02:18:33 -07:00
+8 5f216fc33f qwen 3.8 rebase (#35758)
Co-authored-by: cherichy <cherichy@outlook.com>
Co-authored-by: guangyunh-nv <guangyunh@nvidia.com>
Co-authored-by: jiahanc <jiahanc@nvidia.com>
Co-authored-by: jinyangyuan-nvidia <joyuan@nvidia.com>
Co-authored-by: Cheng Hang <chang@nvidia.com>
Co-authored-by: Yicheng Qiang <yqiang@nvidia.com>
Co-authored-by: Sam Li <lsam@nvidia.com>
Co-authored-by: Tom-Zheng <tizheng@nvidia.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: xiaoweiw-nv <xiaoweiw@nvidia.com>
Co-authored-by: Zheng Li <lizheng.cs@zju.edu.cn>
Co-authored-by: yizhang2077 <1109276519@qq.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-08-28 20:41:34 -07:00
Cheng WanandClaude Opus 5 fd40a331bf config: a parallel size has one spelling; a patched scope declares its own (#36621)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 12:56:42 -07:00
20621aa14b [Model] Support Ling-3.0-flash (BailingMoeV3) (#33561)
Signed-off-by: JustinTong <justintong0323@gmail.com>
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
Co-authored-by: 得泽 <zhangkaihong.zkh@antgroup.com>
Co-authored-by: 翎悦 <vito.yy@antgroup.com>
Co-authored-by: 羽癫 <yudian.zy@antgroup.com>
Co-authored-by: tiwei.btw <tiwei.btw@antgroup.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
Co-authored-by: 文赋 <zibin.zb@antgroup.com>
Co-authored-by: JustinTong <justintong0323@gmail.com>
2026-08-26 17:27:23 -07:00
27c36368b6 fix(moe): guard FP8 delegate activation params (#36275)
Signed-off-by: jikuixie <jikuixie@gmail.com>
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: shyeh25 <206795756+shyeh25@users.noreply.github.com>
2026-08-26 12:12:21 +00:00
elvischenv 46d9427b91 Fix MXFP8 MoE weight sizing for non-gated models (#36097) 2026-08-25 22:09:09 +08:00
guzekai01 716a6bf10c feat(humming): support native W4AFP8 checkpoint schemas (#32033) 2026-08-24 18:59:27 +08:00
Sahithi ChigurupatiandMohammad Miadh Angkad 44db041700 [NVIDIA] Fix SM107 MXFP8 activation prep (#35405)
Signed-off-by: Sahithi Chigurupati <chigurupati.sahithi@gmail.com>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
2026-08-23 18:17:03 +08:00
Jimmy Shong 710267dc4c [Quant] Load compressed-tensors kv_cache_scheme scales (#35455) 2026-08-20 19:17:59 +08:00
Jimmy Shong 5375babbac [Quant] Load compressed-tensors quantized lm_head instead of value-casting it (#35228) 2026-08-19 15:37:45 -07:00
YAMY 5f12839591 [Fix] Support Kimi-K3 ModelOpt mixed NVFP4/FP8 checkpoint (#35077) 2026-08-19 08:13:45 -07:00
Enrique Shockwave 92b1d382c7 [Fix] Correct dense FP8 Marlin bias ordering (#35020) 2026-08-17 03:43:36 +00:00
Mohammad Miadh AngkadandMohammad Angkad 6ab4b99bc2 [Quantization] Fix GPTQ scheme attachment broken by LinearBase.scheme default (#34962)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-08-16 00:48:02 -07:00
Spandan Tiwari f68517f644 [AMD][Quantization][Bugfix] Fix bug related to fp8 max on gfx95x for per-token-group quant (ROCm) (#30900) 2026-08-15 19:54:16 -07:00
Colin Z bc7e3ba66c [AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs (#29328) 2026-08-14 21:59:39 -07:00
Carrie ChenandBrayden Zhong 6a5a9eccaa add flashinfer cute-dsl backend for mxfp8 gemm (#34042)
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
2026-08-13 08:50:01 +08:00
fde9ad2531 [Feature] Add Muse Glimmer model support (#34262)
Co-authored-by: sglang-bot <232288953+sglang-bot@users.noreply.github.com>
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-08-11 15:41:52 -07:00
Cheng Wan e216c2bc59 config: the runner and scheduler read resolved config from the bags (#34095) 2026-08-09 14:45:11 -07:00
Ziang LiandBrayden Zhong 4ad990ba7d [ModelOpt FP4] Support online MoE weight quantization (#33115)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2026-08-06 11:01:55 -07:00