minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

chore: add an unified server arg for multimodal inputs preprocess config(#12149)

Browse Source 
Co-authored-by: bianfeng <bianfeng@pinduoduo.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
This commit is contained in:
wingedge
2025-11-18 12:18:50 +08:00
committed by GitHub
co-authored by bianfeng Xinyuan Tong
parent aa8ecbda7a
commit f1be8aa0f2
4 changed files with 38 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/supported_models/multimodal_language_models.md
+6
View File
@@ -101,3 +101,9 @@ For multimodal models, you can use the `--keep-mm-feature-on-device` flag to opt
- **With `--keep-mm-feature-on-device`**: Feature tensors remain on GPU, reducing device-to-host copy overhead and improving latency, but consuming more GPU memory
Use this flag when you have sufficient GPU memory and want to minimize latency for multimodal inference.
### Multimodal Inputs Limitation
- **Use `--mm-process-config '{"image":{"max_pixels":1048576},"video":{"fps":3,"max_pixels":602112,"max_frames":60}}'`**: To set `image`, `video`, and `audio` input limits.
This can reduce GPU memory usage, improve inference speed, and help to avoid OOM, but may impact model performance, thus set a proper value based on your specific use case. Currently, only `qwen_vl` supports this config. Please refer to [qwen_vl processor](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/multimodal/processors/qwen_vl.py) for understanding the meaning of each parameter.