 Mick
|
191f5c7795
|
diffusion: correct check-changes for multimodal_gen (#13375)
|
2025-11-16 20:32:33 +08:00 |
|
|
Mick
|
922525ee6d
|
fix: fix serve command without diffusion dependency (#13246)
|
2025-11-14 10:24:18 +08:00 |
|
|
Mick
|
33d1aeb07f
|
diffusion: refactor task type of models (#13118)
|
2025-11-12 12:39:34 +08:00 |
|
|
Mick
|
60cb716720
|
[diffusion] log: improve logging while multiprocessing (#12997)
|
2025-11-12 12:08:37 +08:00 |
|
|
Mick
|
2f9952cdbf
|
diffusion: remove unused workflows folder (#13114)
|
2025-11-12 11:51:24 +08:00 |
|
|
Mick
|
6a7322ffbc
|
[diffusion] doc: add support_new_models.md (#13043)
|
2025-11-11 12:49:48 +08:00 |
|
|
Mick
|
ddfcb7c8ab
|
minor: fix notebook bug with new model_info fields added for warmup (#13005)
|
2025-11-11 00:46:12 +08:00 |
|
|
Mick
|
5639145fac
|
diffusion: reduce effort of supporting new model (#12982)
|
2025-11-10 21:20:33 +08:00 |
|
|
Mick
|
95876d75cb
|
chore: include a minimum image for vlms when warming-up (#9528)
|
2025-11-10 14:56:59 +08:00 |
|
|
Mick
|
e123648b36
|
diffusion: fix wan-2.2-TI2V and support sp (#12926)
|
2025-11-10 14:37:57 +08:00 |
|
|
Mick
|
307e7a6128
|
diffusion: fix detected file changes rule in CI (#12943)
|
2025-11-10 13:37:16 +08:00 |
|
|
Mick
|
83f89cc615
|
diffusion: skip full CI suite for multimodal_gen changes (#12940)
|
2025-11-10 10:46:24 +08:00 |
|
|
Mick
|
f5b3ccd9a5
|
feat: basic support for server-level multimodal cache (#10775)
|
2025-11-10 00:27:50 +08:00 |
|
|
MickandXinyuan Tong
|
e316bcacb1
|
update multimodal_gen readme (#12825)
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
|
2025-11-07 09:04:44 -08:00 |
|
|
Mick
|
73e9a2ef5c
|
fix: tiny fix cli (#12744)
|
2025-11-06 14:45:50 +08:00 |
|
|
+9
|
7bc1dae095
|
WIP: initial multimodal-gen support (#12484)
Co-authored-by: yhyang201 <yhyang201@gmail.com>
Co-authored-by: yizhang2077 <1109276519@qq.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: ispobock <ispobaoke@gmail.com>
Co-authored-by: JiLi <leege233@gmail.com>
Co-authored-by: CHEN Xi <78632976+RubiaCx@users.noreply.github.com>
Co-authored-by: laixin <xielx@shanghaitech.edu.cn>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
Co-authored-by: jzhang38 <a1286225768@gmail.com>
Co-authored-by: BrianChen1129 <yongqichcd@gmail.com>
Co-authored-by: Kevin Lin <42618777+kevin314@users.noreply.github.com>
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: rlsu9 <r3su@ucsd.edu>
Co-authored-by: Jinzhe Pan <48981407+eigensystem@users.noreply.github.com>
Co-authored-by: foreverpiano <pianoqwz@qq.com>
Co-authored-by: RandNMR73 <notomatthew31@gmail.com>
Co-authored-by: PorridgeSwim <yz3883@columbia.edu>
Co-authored-by: Jiali Chen <90408393+gary-chenjl@users.noreply.github.com>
|
2025-11-05 12:28:52 -08:00 |
|
|
MickandLianmin Zheng
|
823b442945
|
lang: support direct video inference (#9936)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
|
2025-10-23 18:12:39 -07:00 |
|
|
|
770529a731
|
model: support deepseek-ocr (#11891)
Co-authored-by: yhyang201 <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: yhyang201 <yhyang201@gmail.com>
Co-authored-by: Shi Shuai <126407087+shuaills@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
|
2025-10-24 03:15:17 +08:00 |
|
|
Mick
|
8bd26dd4e6
|
ci: fix night-ci with push retry mechanism (#11765)
|
2025-10-23 11:31:05 -07:00 |
|
|
Mick
|
3e4c7da2f5
|
ci: reduce and refactor vlm ut and combine test files (#11062)
|
2025-10-17 15:24:50 +00:00 |
|
|
MickandXinyuan Tong
|
86b04d25b3
|
model: qwen3-omni (thinker-only) (#10911)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
|
2025-10-16 13:20:38 -07:00 |
|
|
Mick
|
f35f120d70
|
fix: fix video input for qwen3-vl (#11442)
|
2025-10-13 09:30:43 -07:00 |
|
|
Mick
|
0c0779d667
|
ci: improve nightly-ci (#11385)
|
2025-10-12 21:19:34 -07:00 |
|
|
MickandLianmin Zheng
|
9f2b457cbe
|
doc: add doc for adding new models into nightly-ci (#11443)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
|
2025-10-12 08:35:10 -07:00 |
|
|
Mick
|
a1a20b4c7c
|
fix: fix video input for qwen3-vl (#11361)
|
2025-10-10 04:35:35 -07:00 |
|
|
Mick
|
a3c2ea4451
|
fix: fix revision for sgl-flash-attn in sgl-kernel (#11327)
|
2025-10-08 15:50:44 -07:00 |
|
|
Mick
|
64d1505c0a
|
ci: unify the model launch method of nightly ci (#11230)
|
2025-10-07 18:13:14 -07:00 |
|
|
Mick
|
97d966a7f8
|
ci: make find_local_hf_snapshot_dir more robust (#11248)
|
2025-10-05 19:50:11 -07:00 |
|
|
Mick
|
9de1320b63
|
fix: fp8 mllama4 without vision modules being quantized (#10611)
|
2025-09-29 21:17:12 +00:00 |
|
|
Mick
|
43fa9f22bd
|
fix: check if weights are already local before downloading (#11015)
|
2025-09-28 20:11:33 -07:00 |
|
|
Mick
|
2e7633982c
|
fix: show failed models in nightly ci (#10986)
|
2025-09-28 12:38:29 -07:00 |
|
|
Mick
|
777eb53897
|
ci: refactor nightly test (#10495)
|
2025-09-26 15:24:30 -07:00 |
|
|
Mick
|
fff7fbabe6
|
ci: fix rate-limit of huggingface with hf auth login (#10947)
|
2025-09-26 11:02:44 -07:00 |
|
|
Mick
|
0549f21c60
|
fix: fix max_new_tokens uninitialized error (#9343)
|
2025-09-15 12:06:55 +08:00 |
|
|
Mickandyhyang201
|
16a6d21b95
|
chore: enhance bench_serving for vlms with a new dataset of configurable image count and resolution (#9583)
Co-authored-by: yhyang201 <yhyang201@gmail.com>
|
2025-08-26 17:42:54 -07:00 |
|
|
Mick
|
a1f011d09a
|
minor: determine mm attn backend based on platforms (#9303)
|
2025-08-22 01:08:41 -07:00 |
|
|
Mick
|
ef3004d90a
|
misc: parse bench_serving result as markdown table (#9377)
|
2025-08-20 16:44:20 -07:00 |
|
|
Mick
|
1df84ff414
|
ci: simplify multi-modality tests by using mixins (#9006)
|
2025-08-16 22:25:02 -07:00 |
|
|
Mick
|
584e1ab2d0
|
fix: fix unsupported palette mode of images in bench_serving for mmmu (#9206)
|
2025-08-14 18:44:46 -07:00 |
|
|
Mick
|
41d71ca488
|
fix: fix obsolete qwen-audio processor arg (#9003)
|
2025-08-09 13:18:36 -07:00 |
|
|
Mickandzhyncs
|
01c99a9959
|
chore: update Dockerfile (#8872)
Co-authored-by: zhyncs <me@zhyncs.com>
|
2025-08-06 09:30:33 -07:00 |
|
|
MickandXinyuan Tong
|
1992ef9ba7
|
fix: temporarily disable cuda-ipc for mm data tensor (#8431)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
|
2025-07-29 22:42:03 +00:00 |
|
|
Mick
|
0bcc195f4e
|
fix: minor fix TransportProxyTensor under tp (#8382)
|
2025-07-27 00:38:49 -07:00 |
|
|
MickandXinyuan Tong
|
3212c2ad3f
|
vlm: optimize tensor transport (#6003)
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
|
2025-07-26 17:41:01 +08:00 |
|
|
Mick
|
534756749a
|
chore: improvements on mm_utils (#7737)
|
2025-07-26 17:38:56 +08:00 |
|
|
MickandXinyuan Tong
|
4fa44d63c6
|
chore: improve mmmu benchmark (#7000)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
|
2025-07-26 16:19:45 +08:00 |
|
|
Mick
|
3964b352c3
|
chore: tune mem fraction static for vlm (#6881)
|
2025-07-18 17:19:27 -07:00 |
|
|
Mick
|
e1020dc588
|
refactor: simply MultimodalTokens logic (#7924)
|
2025-07-17 17:59:15 -07:00 |
|
|
Mick
|
4395c87a9b
|
refactor: unify names of the feature field of MultimodalDataItem (#8075)
|
2025-07-16 17:52:38 -07:00 |
|
|
Mick
|
497efe747d
|
Revert "feat: replace Decord with video_reader-rs" (#8077)
|
2025-07-15 20:04:56 -07:00 |
|
|
Mick
|
b5e3d6031c
|
vlm: support video as an input modality (#5888)
|
2025-07-09 23:48:35 -07:00 |
|
|
Mick
|
199d621845
|
ci: fix port args (#7792)
|
2025-07-05 15:06:42 -07:00 |
|
|
Mick
|
c797322280
|
fix: fix apply_shuffle_mul_sum (#7444)
|
2025-07-04 23:23:30 -07:00 |
|
|
Mick
|
4d67025a1d
|
chore: improve ci bug reporting (#7542)
|
2025-06-26 01:32:44 -07:00 |
|
|
Mick
|
83d87685c5
|
vlm: adapt internvl to VisionAttention (#6870)
|
2025-06-11 01:16:04 -07:00 |
|
|
Mick
|
ce9d690ef4
|
fix: fix nightly test from updating transformers (#6658)
|
2025-05-27 00:28:11 -07:00 |
|
|
Mick
|
626ccb7d3f
|
vlm: tensor hash kernel (#5974)
|
2025-05-18 15:38:16 -07:00 |
|
|
Mick
|
01dd39bac1
|
refactor: minor refactors regarding multimodal processing (#6187)
|
2025-05-17 22:53:20 -07:00 |
|
|
|
cd7c8a8de6
|
doc: update developer guide regarding mllms (#6138)
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
Co-authored-by: XinyuanTong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <justinning0323@outlook.com>
|
2025-05-14 23:13:13 +08:00 |
|
|
Mick
|
02723e1b0d
|
CI: rewrite test_vision_chunked_prefill to speedup (#5682)
|
2025-04-26 18:33:13 -07:00 |
|
|
|
feda9b11b3
|
fix: fix one more bug from merging mm_inputs (#5718)
Co-authored-by: Xinyuan Tong <justinning0323@outlook.com>
Co-authored-by: XinyuanTong <115166877+JustinTong0323@users.noreply.github.com>
|
2025-04-25 17:28:33 -07:00 |
|
|
MickandXinyuan Tong
|
c998d04b46
|
vlm: enable radix cache for qwen-vl models (#5349)
Co-authored-by: Xinyuan Tong <justinning0323@outlook.com>
|
2025-04-23 20:35:05 -07:00 |
|
|
Mick
|
34ef6c8135
|
[VLM] Adopt fast image processor by default (#5065)
|
2025-04-11 21:46:58 -07:00 |
|
|
Mick
|
e53a0b3d5b
|
[fix] fix mrope positions not picked up (#5265)
|
2025-04-11 01:29:45 -07:00 |
|
|
Mick
|
fbebcb7aa4
|
model: support mllama4 (#5144)
|
2025-04-09 09:28:44 -07:00 |
|
|
Mick
|
5cb552b1d4
|
refactor: multimodal data (#4754)
|
2025-03-31 09:57:51 -07:00 |
|
|
Mick
|
1e86457c90
|
model: Minicpmo (#3023)
|
2025-03-24 20:08:40 -07:00 |
|
|
Mick
|
11577cedb7
|
refactor: bug fixes and refactor for vlm (#4661)
|
2025-03-22 22:48:49 -07:00 |
|
|
Mick
|
d373a48c98
|
fix: second_per_grid_ts should be used to get mrope position (#3682)
|
2025-03-17 18:12:38 -07:00 |
|
|
Mick
|
98be3bd306
|
refactor: rewrite bench-mmmu-sglang (#4458)
|
2025-03-17 18:11:47 -07:00 |
|
|
Mick
|
0f52fb55ec
|
config: Update fused moe config (#4493)
|
2025-03-16 23:51:58 -07:00 |
|
|
Mick
|
9d02bb3e2a
|
Urgent model support: support gemma-3-it (#4424)
|
2025-03-16 17:37:32 -07:00 |
|
|
Mick
|
8ec2ce0726
|
perf: update fused moe config (#4459)
|
2025-03-15 21:23:57 -07:00 |
|
|
Mick
|
035ac2ab74
|
ci: update transformers==4.48.3 (#4451)
|
2025-03-15 13:27:26 -07:00 |
|
|
Mick
|
01090e8ac3
|
model: Support Janus-pro (#3203)
|
2025-03-12 11:02:11 -07:00 |
|
|
Mick
|
ff2ce0b86f
|
refactor: move image processors to separate files (#4229)
|
2025-03-11 12:35:35 -07:00 |
|
|
Mickandzhaochenyang20
|
583d6af71b
|
example: add vlm to token in & out example (#3941)
Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
|
2025-03-04 22:18:26 -08:00 |
|
|
Mick
|
45205d88a0
|
bench: Add MMMU benchmark for vLM (#3562)
|
2025-02-22 08:10:59 -08:00 |
|
|
Mick
|
99c1b9d2ee
|
fix: apply cache size limit of attention mask for VisionAttention (#3657)
|
2025-02-19 20:16:48 +08:00 |
|
|
Mick
|
424848d26f
|
fix: remove dependency on latest transformers impl (#3635)
|
2025-02-19 01:14:11 +08:00 |
|
|
Mick
|
bcc213df61
|
Model: Support Qwen 2.5 vl (#3258)
|
2025-02-16 00:58:53 -08:00 |
|
|
MickandChayenne
|
7711ac6ed0
|
doc: emphasize and notify the usage of chat_template (#3589)
Co-authored-by: Chayenne <zhaochen20@outlook.com>
|
2025-02-15 00:10:32 -08:00 |
|
|
Mickandzhaochenyang20
|
ced680663c
|
doc: Support a new vLM (#3405)
Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
|
2025-02-12 00:43:14 -08:00 |
|
|
Mick
|
9f635ea50d
|
[Fix] Address remaining issues of supporting MiniCPMV (#2977)
|
2025-01-28 00:22:13 -08:00 |
|
|
|
3d93f84a00
|
[Feature] Support minicpmv v2.6 (#2785)
Co-authored-by: Chayenne <zhaochen20@outlook.com>
Co-authored-by: yizhang2077 <1109276519@qq.com>
|
2025-01-18 14:14:19 -08:00 |
|
|
Mick
|
1775b963db
|
[Fix] fix incorrectly overwriting the port specified in ServerArgs (#2714)
|
2025-01-02 10:28:22 -08:00 |
|