vlm: refactor engine vlm params and support processor output as input (#14091)

Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: zhaochenyang20 <zhaochenyang20@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: BenYao21 <cyao22@asu.edu>
Co-authored-by: minleminzui <minleminzui@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: 赵晨阳 <zhaochen20@outlook.com>

docs/advanced_features/vlm_query.ipynb

@@ -5,7 +5,13 @@
    "id": "0",
    "metadata": {},
    "source": [
-    "# Query Vision Language Model"
+    "# Query VLM with Offline Engine\n",
+    "\n",
+    "This tutorial demonstrates how to use SGLang's **offline Engine API** to query VLMs. We will demonstrate usage with Qwen2.5-VL and Llama 4. This section demonstrates three different calling approaches:\n",
+    "\n",
+    "1. **Basic Call**: Directly pass images and text.\n",
+    "2. **Processor Output**: Use HuggingFace processor for data preprocessing.\n",
+    "3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency."
    ]
   },
   {
@@ -13,22 +19,38 @@
    "id": "1",
    "metadata": {},
    "source": [
-    "## Querying Qwen-VL"
+    "## Understanding the Three Input Formats\n",
+    "\n",
+    "SGLang supports three ways to pass visual data, each optimized for different scenarios:\n",
+    "\n",
+    "### 1. **Raw Images** - Simplest approach\n",
+    "- Pass PIL Images, file paths, URLs, or base64 strings directly\n",
+    "- SGLang handles all preprocessing automatically\n",
+    "- Best for: Quick prototyping, simple applications\n",
+    "\n",
+    "### 2. **Processor Output** - For custom preprocessing\n",
+    "- Pre-process images with HuggingFace processor\n",
+    "- Pass the complete processor output dict with `format: \"processor_output\"`\n",
+    "- Best for: Custom image transformations, integration with existing pipelines\n",
+    "- Requirement: Must use `input_ids` instead of text prompt\n",
+    "\n",
+    "### 3. **Precomputed Embeddings** - For maximum performance\n",
+    "- Pre-calculate visual embeddings using the vision encoder\n",
+    "- Pass embeddings with `format: \"precomputed_embedding\"`\n",
+    "- Best for: Repeated queries on same images, caching, high-throughput serving\n",
+    "- Performance gain: Avoids redundant vision encoder computation (30-50% speedup)\n",
+    "\n",
+    "**Key Rule**: Within a single request, use only one format for all images. Don't mix formats.\n",
+    "\n",
+    "The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models."
    ]
   },
   {
-   "cell_type": "code",
+   "cell_type": "markdown",
-   "execution_count": null,
    "id": "2",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "import nest_asyncio\n",
+    "## Querying Qwen2.5-VL Model"
-    "\n",
-    "nest_asyncio.apply()  # Run this first.\n",
-    "\n",
-    "model_path = \"Qwen/Qwen2.5-VL-3B-Instruct\"\n",
-    "chat_template = \"qwen2-vl\""
    ]
   },
   {
@@ -38,8 +60,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Lets create a prompt.\n",
+    "import nest_asyncio\n",
     "\n",
+    "nest_asyncio.apply()\n",
+    "\n",
+    "model_path = \"Qwen/Qwen2.5-VL-3B-Instruct\"\n",
+    "chat_template = \"qwen2-vl\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "from io import BytesIO\n",
     "import requests\n",
     "from PIL import Image\n",
@@ -59,30 +94,18 @@
     "conv.append_message(conv.roles[1], \"\")\n",
     "conv.image_data = [image]\n",
     "\n",
+    "print(\"Generated prompt text:\")\n",
     "print(conv.get_prompt())\n",
+    "print(f\"\\nImage size: {image.size}\")\n",
     "image"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4",
-   "metadata": {},
-   "source": [
-    "### Query via the offline Engine API"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
    "id": "5",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "from sglang import Engine\n",
+    "### Basic Offline Engine API Call"
-    "\n",
-    "llm = Engine(\n",
-    "    model_path=model_path, chat_template=chat_template, mem_fraction_static=0.8\n",
-    ")"
    ]
   },
   {
@@ -92,27 +115,73 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "out = llm.generate(prompt=conv.get_prompt(), image_data=[image])\n",
+    "from sglang import Engine\n",
-    "print(out[\"text\"])"
+    "\n",
-   ]
+    "\n",
-  },
+    "llm = Engine(model_path=model_path, chat_template=chat_template, log_level=\"warning\")"
-  {
-   "cell_type": "markdown",
-   "id": "7",
-   "metadata": {},
-   "source": [
-    "### Query via the offline Engine API, but send precomputed embeddings"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8",
+   "id": "7",
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Compute the image embeddings using Huggingface.\n",
+    "out = llm.generate(prompt=conv.get_prompt(), image_data=[image])\n",
+    "print(\"Model response:\")\n",
+    "print(out[\"text\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8",
+   "metadata": {},
+   "source": [
+    "### Call with Processor Output\n",
     "\n",
+    "Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoProcessor\n",
+    "\n",
+    "processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
+    "processor_output = processor(\n",
+    "    images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
+    ")\n",
+    "\n",
+    "out = llm.generate(\n",
+    "    input_ids=processor_output[\"input_ids\"][0].detach().cpu().tolist(),\n",
+    "    image_data=[dict(processor_output, format=\"processor_output\")],\n",
+    ")\n",
+    "print(\"Response using processor output:\")\n",
+    "print(out[\"text\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10",
+   "metadata": {},
+   "source": [
+    "### Call with Precomputed Embeddings\n",
+    "\n",
+    "You can pre-calculate image features to avoid repeated visual encoding processes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "11",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "from transformers import AutoProcessor\n",
     "from transformers import Qwen2_5_VLForConditionalGeneration\n",
     "\n",
@@ -122,53 +191,6 @@
     ")"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "processed_prompt = processor(\n",
-    "    images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
-    ")\n",
-    "input_ids = processed_prompt[\"input_ids\"][0].detach().cpu().tolist()\n",
-    "precomputed_embeddings = vision(\n",
-    "    processed_prompt[\"pixel_values\"].cuda(), processed_prompt[\"image_grid_thw\"].cuda()\n",
-    ")\n",
-    "\n",
-    "mm_item = dict(\n",
-    "    modality=\"IMAGE\",\n",
-    "    image_grid_thw=processed_prompt[\"image_grid_thw\"],\n",
-    "    precomputed_embeddings=precomputed_embeddings,\n",
-    ")\n",
-    "out = llm.generate(input_ids=input_ids, image_data=[mm_item])\n",
-    "print(out[\"text\"])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "10",
-   "metadata": {},
-   "source": [
-    "## Querying Llama 4 (Vision)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "11",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import nest_asyncio\n",
-    "\n",
-    "nest_asyncio.apply()  # Run this first.\n",
-    "\n",
-    "model_path = \"meta-llama/Llama-4-Scout-17B-16E-Instruct\"\n",
-    "chat_template = \"llama-4\""
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -176,7 +198,39 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Lets create a prompt.\n",
+    "processor_output = processor(\n",
+    "    images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
+    ")\n",
+    "\n",
+    "input_ids = processor_output[\"input_ids\"][0].detach().cpu().tolist()\n",
+    "\n",
+    "precomputed_embeddings = vision(\n",
+    "    processor_output[\"pixel_values\"].cuda(), processor_output[\"image_grid_thw\"].cuda()\n",
+    ")\n",
+    "\n",
+    "multi_modal_item = dict(\n",
+    "    processor_output,\n",
+    "    format=\"precomputed_embedding\",\n",
+    "    feature=precomputed_embeddings,\n",
+    ")\n",
+    "\n",
+    "out = llm.generate(input_ids=input_ids, image_data=[multi_modal_item])\n",
+    "print(\"Response using precomputed embeddings:\")\n",
+    "print(out[\"text\"])\n",
+    "\n",
+    "llm.shutdown()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13",
+   "metadata": {},
+   "source": [
+    "## Querying Llama 4 Vision Model\n",
+    "\n",
+    "```python\n",
+    "model_path = \"meta-llama/Llama-4-Scout-17B-16E-Instruct\"\n",
+    "chat_template = \"llama-4\"\n",
     "\n",
     "from io import BytesIO\n",
     "import requests\n",
@@ -184,6 +238,7 @@
     "\n",
     "from sglang.srt.parser.conversation import chat_templates\n",
     "\n",
+    "# Download the same example image\n",
     "image = Image.open(\n",
     "    BytesIO(\n",
     "        requests.get(\n",
@@ -197,53 +252,62 @@
     "conv.append_message(conv.roles[1], \"\")\n",
     "conv.image_data = [image]\n",
     "\n",
+    "print(\"Llama 4 generated prompt text:\")\n",
     "print(conv.get_prompt())\n",
     "print(f\"Image size: {image.size}\")\n",
     "\n",
-    "image"
+    "image\n",
+    "```"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "13",
-   "metadata": {},
-   "source": [
-    "### Query via the offline Engine API"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
    "id": "14",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "from sglang.test.test_utils import is_in_ci\n",
+    "### Llama 4 Basic Call\n",
     "\n",
-    "if not is_in_ci():\n",
+    "Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp_size=4) and larger context length.\n",
-    "    from sglang import Engine\n",
     "\n",
-    "    llm = Engine(\n",
+    "```python\n",
+    "llm = Engine(\n",
     "    model_path=model_path,\n",
-    "        trust_remote_code=True,\n",
     "    enable_multimodal=True,\n",
-    "        mem_fraction_static=0.8,\n",
-    "        tp_size=4,\n",
     "    attention_backend=\"fa3\",\n",
+    "    tp_size=4,\n",
     "    context_length=65536,\n",
-    "    )"
+    ")\n",
+    "\n",
+    "out = llm.generate(prompt=conv.get_prompt(), image_data=[image])\n",
+    "print(\"Llama 4 response:\")\n",
+    "print(out[\"text\"])\n",
+    "```"
    ]
   },
   {
-   "cell_type": "code",
+   "cell_type": "markdown",
-   "execution_count": null,
    "id": "15",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "if not is_in_ci():\n",
+    "### Call with Processor Output\n",
-    "    out = llm.generate(prompt=conv.get_prompt(), image_data=[image])\n",
+    "\n",
-    "    print(out[\"text\"])"
+    "Using HuggingFace processor to preprocess data can reduce computational overhead during inference.\n",
+    "\n",
+    "```python\n",
+    "from transformers import AutoProcessor\n",
+    "\n",
+    "processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
+    "processor_output = processor(\n",
+    "    images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
+    ")\n",
+    "\n",
+    "out = llm.generate(\n",
+    "    input_ids=processor_output[\"input_ids\"][0].detach().cpu().tolist(),\n",
+    "    image_data=[dict(processor_output, format=\"processor_output\")],\n",
+    ")\n",
+    "print(\"Response using processor output:\")\n",
+    "print(out)\n",
+    "```"
    ]
   },
   {
@@ -251,54 +315,48 @@
    "id": "16",
    "metadata": {},
    "source": [
-    "### Query via the offline Engine API, but send precomputed embeddings"
+    "### Call with Precomputed Embeddings\n",
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "17",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "if not is_in_ci():\n",
-    "    # Compute the image embeddings using Huggingface.\n",
     "\n",
-    "    from transformers import AutoProcessor\n",
+    "```python\n",
-    "    from transformers import Llama4ForConditionalGeneration\n",
+    "from transformers import AutoProcessor\n",
+    "from transformers import Llama4ForConditionalGeneration\n",
     "\n",
-    "    processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
+    "processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
-    "    model = Llama4ForConditionalGeneration.from_pretrained(\n",
+    "model = Llama4ForConditionalGeneration.from_pretrained(\n",
     "    model_path, torch_dtype=\"auto\"\n",
-    "    ).eval()\n",
+    ").eval()\n",
-    "    vision = model.vision_model.cuda()\n",
-    "    multi_modal_projector = model.multi_modal_projector.cuda()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "18",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "if not is_in_ci():\n",
-    "    processed_prompt = processor(\n",
-    "        images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
-    "    )\n",
-    "    print(f'{processed_prompt[\"pixel_values\"].shape=}')\n",
-    "    input_ids = processed_prompt[\"input_ids\"][0].detach().cpu().tolist()\n",
     "\n",
-    "    image_outputs = vision(\n",
+    "vision = model.vision_model.cuda()\n",
-    "        processed_prompt[\"pixel_values\"].to(\"cuda\"), output_hidden_states=False\n",
+    "multi_modal_projector = model.multi_modal_projector.cuda()\n",
-    "    )\n",
-    "    image_features = image_outputs.last_hidden_state\n",
-    "    vision_flat = image_features.view(-1, image_features.size(-1))\n",
-    "    precomputed_embeddings = multi_modal_projector(vision_flat)\n",
     "\n",
-    "    mm_item = dict(modality=\"IMAGE\", precomputed_embeddings=precomputed_embeddings)\n",
+    "print(f'Image pixel values shape: {processor_output[\"pixel_values\"].shape}')\n",
-    "    out = llm.generate(input_ids=input_ids, image_data=[mm_item])\n",
+    "input_ids = processor_output[\"input_ids\"][0].detach().cpu().tolist()\n",
-    "    print(out[\"text\"])"
+    "\n",
+    "# Process image through vision encoder\n",
+    "image_outputs = vision(\n",
+    "    processor_output[\"pixel_values\"].to(\"cuda\"), \n",
+    "    aspect_ratio_ids=processor_output[\"aspect_ratio_ids\"].to(\"cuda\"),\n",
+    "    aspect_ratio_mask=processor_output[\"aspect_ratio_mask\"].to(\"cuda\"),\n",
+    "    output_hidden_states=False\n",
+    ")\n",
+    "image_features = image_outputs.last_hidden_state\n",
+    "\n",
+    "# Flatten image features and pass through multimodal projector\n",
+    "vision_flat = image_features.view(-1, image_features.size(-1))\n",
+    "precomputed_embeddings = multi_modal_projector(vision_flat)\n",
+    "\n",
+    "# Build precomputed embedding data item\n",
+    "mm_item = dict(\n",
+    "    processor_output, \n",
+    "    format=\"precomputed_embedding\", \n",
+    "    feature=precomputed_embeddings\n",
+    ")\n",
+    "\n",
+    "# Use precomputed embeddings for efficient inference\n",
+    "out = llm.generate(input_ids=input_ids, image_data=[mm_item])\n",
+    "print(\"Llama 4 precomputed embedding response:\")\n",
+    "print(out[\"text\"])\n",
+    "```"
    ]
   }
  ],
@@ -306,7 +364,13 @@
   "jupytext": {
    "cell_metadata_filter": "-all",
    "custom_cell_magics": "kql",
-   "encoding": "# -*- coding: utf-8 -*-"
+   "encoding": "# -*- coding: utf-8 -*-",
+   "text_representation": {
+    "extension": ".py",
+    "format_name": "light",
+    "format_version": "1.5",
+    "jupytext_version": "1.16.1"
+   }
   },
   "language_info": {
    "codemirror_mode": {

docs/basic_usage/sampling_params.md

@@ -12,7 +12,7 @@ The `/generate` endpoint accepts the following parameters in JSON format. For de
 | text                       | `Optional[Union[List[str], str]] = None`                                     | The input prompt. Can be a single prompt or a batch of prompts.                                                                                                 |
 | input_ids                  | `Optional[Union[List[List[int]], List[int]]] = None`                         | The token IDs for text; one can specify either text or input_ids.                                                                                               |
 | input_embeds               | `Optional[Union[List[List[List[float]]], List[List[float]]]] = None`         | The embeddings for input_ids; one can specify either text, input_ids, or input_embeds.                                                                          |
-| image_data                 | `Optional[Union[List[List[ImageDataItem]], List[ImageDataItem], ImageDataItem]] = None` | The image input. Can be an image instance, file name, URL, or base64 encoded string. Can be a single image, list of images, or list of lists of images. |
+| image_data                 | `Optional[Union[List[List[ImageDataItem]], List[ImageDataItem], ImageDataItem]] = None` | The image input. Supports three formats: (1) **Raw images**: PIL Image, file path, URL, or base64 string; (2) **Processor output**: Dict with `format: "processor_output"` containing HuggingFace processor outputs; (3) **Precomputed embeddings**: Dict with `format: "precomputed_embedding"` and `feature` containing pre-calculated visual embeddings. Can be a single image, list of images, or list of lists of images. See [Multimodal Input Formats](#multimodal-input-formats) for details. |
 | audio_data                 | `Optional[Union[List[AudioDataItem], AudioDataItem]] = None`                 | The audio input. Can be a file name, URL, or base64 encoded string.                                                                                             |
 | sampling_params            | `Optional[Union[List[Dict], Dict]] = None`                                   | The sampling parameters as described in the sections below.                                                                                                     |
 | rid                        | `Optional[Union[List[str], str]] = None`                                     | The request ID.                                                                                                                                                 |

python/sglang/srt/entrypoints/engine.py

@@ -273,6 +273,8 @@ class Engine(EngineBase):
         # - Single image for a single request
         # - List of images (one per request in a batch)
         # - List of lists of images (multiple images per request)
+        # - List of preprocessed outputs from a Huggingface processor, each as a dict containing `format`: 'processor_output' and other data
+        # - List of precomputed image embeddings, each as a dict containing field `format`: 'precomputed_embedding' and `feature`: the precomputed embedding
         # See also python/sglang/srt/utils.py:load_image for more details.
         image_data: Optional[MultimodalDataInputFormat] = None,
         audio_data: Optional[MultimodalDataInputFormat] = None,
@@ -355,6 +357,8 @@ class Engine(EngineBase):
         # - Single image for a single request
         # - List of images (one per request in a batch)
         # - List of lists of images (multiple images per request)
+        # - List of preprocessed outputs from a Huggingface processor, each as a dict containing `format`: 'processor_output' and other data
+        # - List of precomputed image embeddings, each as a dict containing field `format`: 'precomputed_embedding' and `feature`: the precomputed embedding
         # See also python/sglang/srt/utils.py:load_image for more details.
         image_data: Optional[MultimodalDataInputFormat] = None,
         audio_data: Optional[MultimodalDataInputFormat] = None,

python/sglang/srt/function_call/llama32_detector.py

@@ -1,5 +1,7 @@
+import ast
 import json
 import logging
+import re
 from typing import List
 
 from sglang.srt.entrypoints.openai.protocol import Tool
@@ -32,6 +34,16 @@ class Llama32Detector(BaseFormatDetector):
         # if users define to use a different separator in their prompt
         self.tool_call_separator = ";"
 
+    def _convert_python_dict_to_json(self, text: str) -> str:
+        """Convert Python dict strings to JSON format."""
+        try:
+            parsed = ast.literal_eval(text.strip())
+            if isinstance(parsed, dict):
+                return json.dumps(parsed, ensure_ascii=False)
+        except:
+            pass
+        return text
+
     def has_tool_call(self, text: str) -> bool:
         """Check if the text contains a Llama 3.2 format tool call."""
         # depending on the prompt format the Llama model may or may not
@@ -59,16 +71,36 @@ class Llama32Detector(BaseFormatDetector):
                 all_actions.append(obj)
                 idx += end + len(self.tool_call_separator)
                 safe_idx = idx
-            except json.JSONDecodeError as e:
+            except json.JSONDecodeError:
-                # Find where next `{"name"` appears and try again
+                # Try Python dict conversion as fallback
-                logger.warning(
+                try:
-                    f"Failed to parse JSON part: {action_text[idx:]}, JSON parse error: {str(e)}"
+                    dict_end = idx
-                )
+                    brace_count = 0
+                    for i in range(idx, action_text_len):
+                        if action_text[i] == "{":
+                            brace_count += 1
+                        elif action_text[i] == "}":
+                            brace_count -= 1
+                            if brace_count == 0:
+                                dict_end = i + 1
+                                break
+
+                    if dict_end > idx:
+                        potential_dict = action_text[idx:dict_end]
+                        json_version = self._convert_python_dict_to_json(potential_dict)
+                        if json_version != potential_dict:
+                            obj, _ = decoder.raw_decode(json_version)
+                            all_actions.append(obj)
+                            idx = dict_end + len(self.tool_call_separator)
+                            safe_idx = idx
+                            continue
+                except:
+                    pass
+
                 next_obj_start = action_text.find('{"name":', idx + 1)
                 if next_obj_start == -1:
                     break
                 idx = next_obj_start
-                continue
 
         # Only process if we found valid JSON objects
         calls = self.parse_base_json(all_actions, tools) if all_actions else []
@@ -80,6 +112,30 @@ class Llama32Detector(BaseFormatDetector):
             normal_text=normal_text + trailing_text, calls=calls
         )
 
+    def parse_streaming_increment(
+        self, new_text: str, tools: List[Tool]
+    ) -> StreamingParseResult:
+        """Override to handle Python dict format in streaming."""
+        # First try with converted Python dict
+        self._buffer += new_text
+        converted_buffer = self._buffer
+
+        # Convert Python dict syntax to JSON
+        converted_buffer = re.sub(r"'([^']*)':", r'"\1":', converted_buffer)
+        converted_buffer = re.sub(r":\s*'([^']*)'", r': "\1"', converted_buffer)
+
+        # Temporarily replace buffer for parsing
+        original_buffer = self._buffer
+        self._buffer = converted_buffer
+
+        try:
+            result = super().parse_streaming_increment("", tools)
+            return result
+        except:
+            # Fall back to original buffer
+            self._buffer = original_buffer
+            return super().parse_streaming_increment(new_text, tools)
+
     def structure_info(self) -> _GetInfoFunc:
         return lambda name: StructureInfo(
             begin='<|python_tag|>{"name":"' + name + '", "arguments":',

python/sglang/srt/managers/schedule_batch.py

@@ -189,6 +189,12 @@ class Modality(Enum):
         return [Modality.IMAGE, Modality.VIDEO, Modality.AUDIO]
 
 
+class MultimodalInputFormat(Enum):
+    NORMAL = auto()
+    PROCESSOR_OUTPUT = auto()
+    PRECOMPUTED_EMBEDDING = auto()
+
+
 @dataclasses.dataclass
 class MultimodalDataItem:
     """
@@ -204,6 +210,8 @@ class MultimodalDataItem:
     pad_value: int = None
     offsets: Optional[list] = None
 
+    format: MultimodalInputFormat = MultimodalInputFormat.NORMAL
+
     # the raw features returned by processor, e.g. pixel_values or audio_features
     feature: Union[torch.Tensor, np.ndarray] = None
     # the precomputed embeddings, passed as final encoder embeddings
@@ -276,6 +284,9 @@ class MultimodalDataItem:
         ...
         # TODO
 
+    def is_precomputed_embedding(self):
+        return self.format == MultimodalInputFormat.PRECOMPUTED_EMBEDDING
+
     @staticmethod
     def from_dict(obj: dict):
         kwargs = dict(obj)

python/sglang/srt/models/gemma3_causal.py

@@ -373,14 +373,20 @@ class Gemma3RotaryEmbedding(nn.Module):
         # BC: "rope_type" was originally "type"
         if hasattr(config, "rope_scaling") and config.rope_scaling is not None:
             self.rope_type = config.rope_scaling.get(
-                "rope_type", config.rope_scaling.get("type")
+                "rope_type", config.rope_scaling.get("type", "default")
             )
+
         else:
             self.rope_type = "default"
+
+        if self.rope_type is None:
+            self.rope_type = "default"
+
         self.max_seq_len_cached = config.max_position_embeddings
         self.original_max_seq_len = config.max_position_embeddings
 
         self.config = config
+
         self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
 
         inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device)

python/sglang/srt/models/gemma3_mm.py

@@ -290,15 +290,26 @@ class Gemma3ForConditionalGeneration(PreTrainedModel):
     def get_image_feature(self, items: List[MultimodalDataItem]):
         """
         Projects the last hidden state from the vision model into language model space.
+        Supports both raw image pixel values and precomputed embeddings.
 
         Returns:
             image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`).
         """
         # Process images one by one to handle flatten_batch=True constraint in vision_tower
         all_pixel_values = flatten_nested_list([item.feature for item in items])
-        vision_outputs_list = []
+
+        final_features_list = []
 
         for pixel_values_batch in all_pixel_values:
+            if (
+                pixel_values_batch.dim() == 3
+                and pixel_values_batch.shape[-1] == self.config.text_config.hidden_size
+            ):
+                final_features_list.append(
+                    pixel_values_batch.to(self.language_model.device)
+                )
+                continue
+
             # Normalize input shape to [batch_size, channels, height, width]
             if pixel_values_batch.dim() == 5:
                 pixel_values_batch = pixel_values_batch.squeeze(0)
@@ -309,20 +320,29 @@ class Gemma3ForConditionalGeneration(PreTrainedModel):
                     f"Unexpected pixel_values shape: {pixel_values_batch.shape}"
                 )
 
-            # Process each image in the batch
+            # Process each image in the batch through Vision Tower
+            batch_vision_outputs = []
             batch_size = pixel_values_batch.shape[0]
+
             for i in range(batch_size):
                 pixel_value = pixel_values_batch[i : i + 1]  # Keep batch dimension as 1
                 pixel_value = pixel_value.to(
                     device=self.vision_tower.device, dtype=self.language_model.dtype()
                 )
                 vision_output = self.vision_tower(pixel_values=pixel_value)
-                vision_outputs_list.append(vision_output)
+                batch_vision_outputs.append(vision_output)
 
-        # Concatenate all vision outputs
+            if batch_vision_outputs:
-        vision_outputs = torch.cat(vision_outputs_list, dim=0)
+                vision_outputs_cat = torch.cat(batch_vision_outputs, dim=0)
-        image_features = self.multi_modal_projector(vision_outputs)
+
-        return image_features
+                projected_features = self.multi_modal_projector(vision_outputs_cat)
+                final_features_list.append(projected_features)
+
+        # Concatenate all features (all are now in text space)
+        if final_features_list:
+            return torch.cat(final_features_list, dim=0)
+        else:
+            return torch.tensor([], device=self.language_model.device)
 
     @torch.no_grad()
     def forward(

python/sglang/srt/models/kimi_vl.py

@@ -142,6 +142,13 @@ class KimiVLForConditionalGeneration(nn.Module):
             .type(self.vision_tower.dtype)
             .to(self.vision_tower.device)
         )
+
+        if (
+            pixel_values.dim() == 2
+            and pixel_values.shape[-1] == self.config.text_config.hidden_size
+        ):
+            return pixel_values
+
         image_grid_hws = torch.cat([item.image_grid_hws for item in items], dim=0).to(
             self.vision_tower.device
         )

python/sglang/srt/models/minicpmo.py

@@ -1668,6 +1668,24 @@ class MiniCPMO(MiniCPMBaseModel):
             [item.audio_feature_lens for item in items if item.audio_feature_lens]
         )
 
+        # Ensure audio_feature_lens_raw is properly formatted as [[tensor], [tensor], ...]
+        if audio_feature_lens_raw:
+            if isinstance(audio_feature_lens_raw[0], torch.Tensor):
+                # Flat list of tensors, wrap each in a list
+                audio_feature_lens_raw = [[lens] for lens in audio_feature_lens_raw]
+            elif isinstance(audio_feature_lens_raw[0], list):
+                # Already nested, ensure all elements are properly formatted
+                # Flatten if needed
+                flattened = []
+                for item in audio_feature_lens_raw:
+                    if isinstance(item, list):
+                        flattened.extend(item)
+                    else:
+                        flattened.append(item)
+                audio_feature_lens_raw = [
+                    [item] if not isinstance(item, list) else item for item in flattened
+                ]
+
         final_audio_embeds = []
 
         assert isinstance(wavforms, list)
@@ -1675,7 +1693,14 @@ class MiniCPMO(MiniCPMBaseModel):
         # exist audio
         for wavform in wavforms:
             if len(wavform) > 0:
-                audio_feature_lens = torch.hstack(audio_feature_lens_raw)
+                # Flatten audio_feature_lens_raw to get a list of tensors
+                flattened_lens = []
+                for item in audio_feature_lens_raw:
+                    if isinstance(item, list):
+                        flattened_lens.extend(item)
+                    else:
+                        flattened_lens.append(item)
+                audio_feature_lens = torch.hstack(flattened_lens)
                 batch_size, _, max_mel_seq_len = wavform.shape
                 max_seq_len = (max_mel_seq_len - 1) // 2 + 1
 

python/sglang/srt/models/qwen2_5_vl.py

@@ -447,7 +447,10 @@ class Qwen2_5_VisionTransformer(nn.Module, RotaryPosMixin):
         # transformers
         x = x.unsqueeze(1)
         for layer_num, blk in enumerate(self.blocks):
-            if layer_num in self.fullatt_block_indexes:
+            fullatt_indexes = self.fullatt_block_indexes
+            if isinstance(fullatt_indexes, torch.Tensor):
+                fullatt_indexes = fullatt_indexes.tolist()
+            if layer_num in fullatt_indexes:
                 cu_seqlens_now = cu_seqlens
             else:
                 cu_seqlens_now = cu_window_seqlens
@@ -630,6 +633,25 @@ class Qwen2_5_VLForConditionalGeneration(nn.Module):
             self.visual.dtype
         )
         image_grid_thw = torch.concat([item.image_grid_thw for item in items], dim=0)
+
+        expected_dim = getattr(self.visual, "embed_dim", -1)
+
+        if expected_dim == -1:
+            vision_conf = self.config.vision_config
+            expected_dim = getattr(
+                vision_conf, "embed_dim", getattr(vision_conf, "hidden_size", -1)
+            )
+
+        raw_patch_dim = 1176
+
+        if pixel_values.dim() == 2:
+            current_dim = pixel_values.shape[-1]
+            if current_dim == expected_dim:
+                return pixel_values
+            if current_dim != raw_patch_dim:
+
+                return pixel_values
+
         assert pixel_values.dim() == 2, pixel_values.dim()
         assert image_grid_thw.dim() == 2, image_grid_thw.dim()
         if self.use_data_parallel:

python/sglang/srt/multimodal/processors/base_processor.py

@@ -12,9 +12,12 @@ import torch
 from PIL import Image
 from transformers import BaseImageProcessorFast
 
-from sglang.srt.environ import envs
+from sglang.srt.managers.schedule_batch import (
-from sglang.srt.managers.schedule_batch import Modality, MultimodalDataItem
+    Modality,
-from sglang.srt.utils import is_npu, load_audio, load_image, load_video, logger
+    MultimodalDataItem,
+    MultimodalInputFormat,
+)
+from sglang.srt.utils import envs, is_npu, load_audio, load_image, load_video, logger
 from sglang.srt.utils.cuda_ipc_transport_utils import (
     MM_FEATURE_CACHE_SIZE,
     MM_ITEM_MEMORY_POOL_RECYCLE_INTERVAL,
@@ -29,7 +32,7 @@ SGL_USE_CUDA_IPC = envs.SGLANG_USE_CUDA_IPC_TRANSPORT.get()
 
 @dataclasses.dataclass
 class BaseMultiModalProcessorOutput:
-    # input_text, with each frame of video/image represented with a image_token
+    # input_text with all multimodality placeholder token expanded
     input_text: str
 
     # frames loaded from image, in given order
@@ -385,10 +388,17 @@ class BaseMultimodalProcessor(ABC):
         """
         Load a single multimodal data.
 
-        If data is precomputed, returns directly.
+        If data is processor_output or precomputed embedding, return directly.
 
         Static method that can be pickled for multiprocessing"""
         if isinstance(data, dict):
+            data_format = data.get("format")
+            if data_format in (
+                MultimodalInputFormat.PROCESSOR_OUTPUT.name,
+                MultimodalInputFormat.PRECOMPUTED_EMBEDDING.name,
+                "processor_output",
+                "precomputed_embedding",
+            ):
                 return data
         try:
             if modality == Modality.IMAGE:
@@ -431,9 +441,10 @@ class BaseMultimodalProcessor(ABC):
                 try:
                     data = next(data_iterator)
                 except StopIteration:
-                    raise ValueError(
+                    logger.warning(
-                        f"Mismatch: More '{text_part}' tokens found than corresponding data items provided."
+                        f"Mismatch: More '{modality.name}' tokens found than corresponding data provided."
                     )
+                    return futures, task_info
 
                 frame_count_limit = None
                 if modality == Modality.IMAGE and image_estimated_frames_iter:
@@ -475,6 +486,77 @@ class BaseMultimodalProcessor(ABC):
 
         return futures, task_info
 
+    @staticmethod
+    def _validate_one_modality(modality: Modality, data_list: Optional[list]):
+        if data_list is None:
+            return
+        if not isinstance(data_list, list):
+            raise TypeError(
+                f"{modality.name} must be a list or None, got {type(data_list)}"
+            )
+
+        formatted_indices = []
+        for idx, item in enumerate(data_list):
+            if isinstance(item, dict):
+                fmt = item.get("format")
+                if fmt in {"processor_output", "precomputed_embedding"}:
+                    formatted_indices.append(idx)
+
+        if formatted_indices:
+            if len(data_list) != 1:
+                raise ValueError(
+                    f"For {modality}, when providing a 'processor_output' or "
+                    f"'precomputed_embedding', you must pass exactly one item; "
+                    f"received {len(data_list)} items (formatted at indices {formatted_indices})."
+                )
+
+    @staticmethod
+    def validate_mm_data(
+        image_data: Optional[list] = None,
+        video_data: Optional[list] = None,
+        audio_data: Optional[list] = None,
+    ):
+        """
+        Validate multimodal input lists per modality.
+
+        Rule per modality (image/video/audio):
+        - Either the list has exactly one item and that single item is a dict with
+          format in {"processor_output", "precomputed_embedding"};
+        - Or, the list contains only "normal" items (i.e., does not include any
+          item whose format is one of the two above).
+
+        Empty or None lists are considered valid.
+        """
+
+        BaseMultimodalProcessor._validate_one_modality(Modality.IMAGE, image_data)
+        BaseMultimodalProcessor._validate_one_modality(Modality.VIDEO, video_data)
+        BaseMultimodalProcessor._validate_one_modality(Modality.AUDIO, audio_data)
+
+    def _process_loaded_mm_data(self, modality, raw_data, result):
+        images, videos, audios = [], [], []
+
+        is_precomputed = isinstance(raw_data, dict) and raw_data.get("format") in [
+            MultimodalInputFormat.PROCESSOR_OUTPUT.name,
+            MultimodalInputFormat.PRECOMPUTED_EMBEDDING.name,
+            "processor_output",
+            "precomputed_embedding",
+        ]
+
+        if modality == Modality.IMAGE:
+            if is_precomputed:
+                images.append(result)
+            else:
+                if isinstance(result, list):
+                    images.extend(result)
+                else:
+                    images.append(result)
+        elif modality == Modality.VIDEO:
+            videos.append(result)
+        elif modality == Modality.AUDIO:
+            audios.append(result)
+
+        return is_precomputed, images, videos, audios
+
     def load_mm_data(
         self,
         prompt: str,
@@ -495,8 +577,10 @@ class BaseMultimodalProcessor(ABC):
             discard_alpha_channel: if True, discards the alpha channel in the returned images
 
         """
-        multimodal_tokens_pattern = multimodal_tokens.get_combined_regex()
 
+        BaseMultimodalProcessor.validate_mm_data(image_data, video_data, audio_data)
+
+        multimodal_tokens_pattern = multimodal_tokens.get_combined_regex()
         if isinstance(prompt, list) and return_text:
             assert len(prompt) and isinstance(prompt[0], int)
             prompt = self._processor.tokenizer.decode(prompt)
@@ -506,7 +590,6 @@ class BaseMultimodalProcessor(ABC):
         assert isinstance(prompt, str)
         # split text into list of normal text and special tokens
         text_parts = re.split(multimodal_tokens_pattern, prompt)
-
         # collect all data
         data_iterators = {}
         if multimodal_tokens.image_token and image_data:
@@ -531,29 +614,31 @@ class BaseMultimodalProcessor(ABC):
         # Process results
         images, videos, audios = [], [], []
         new_text_parts = []
+        has_precomputed_input = False
         for text_part in text_parts:
             try:
                 if multimodal_tokens_pattern.match(text_part):
                     modality, raw_data, frame_limit = next(task_info_iter)
-                    is_precomputed = isinstance(raw_data, dict)
                     result = next(futures_iter).result()
 
-                    if modality == Modality.IMAGE:
+                    is_precomputed, new_imgs, new_vids, new_auds = (
-                        # If data is already processed it will be a
+                        self._process_loaded_mm_data(modality, raw_data, result)
-                        # dictionary(precomputed). In this case we want to keep the
-                        # expanded tokens in text_part. Otherwise, we will
-                        # call the processor code, so keep only a single image
-                        # token.
-                        mm_tokens = (
-                            text_part
-                            if is_precomputed
-                            else multimodal_tokens.image_token
                     )
-                        frames = [result] if not isinstance(result, list) else result
+
-                        if frames:
+                    has_precomputed_input |= is_precomputed
-                            # only for minicpmv
+                    images.extend(new_imgs)
-                            images += frames
+                    videos.extend(new_vids)
-                            new_text_parts += mm_tokens * len(frames)
+                    audios.extend(new_auds)
+
+                    if modality == Modality.IMAGE:
+                        if is_precomputed:
+                            new_text_parts += [text_part]
+                        else:
+                            count = len(new_imgs)
+                            if count > 0:
+                                new_text_parts += [
+                                    multimodal_tokens.image_token
+                                ] * count
                     elif modality == Modality.VIDEO:
                         # load as video
                         mm_tokens = (
@@ -561,7 +646,6 @@ class BaseMultimodalProcessor(ABC):
                             if is_precomputed
                             else multimodal_tokens.video_token
                         )
-                        videos += [result]
                         new_text_parts += mm_tokens
                     elif modality == Modality.AUDIO:
                         # audio
@@ -570,12 +654,19 @@ class BaseMultimodalProcessor(ABC):
                             if is_precomputed
                             else multimodal_tokens.audio_token
                         )
-                        audios += [result]
                         new_text_parts += mm_tokens
                 else:
                     # normal text
                     new_text_parts += [text_part]
 
+            except StopIteration as e:
+                # when precomputed_input is presented with multi-images, StopIteration is expected
+                if has_precomputed_input:
+                    new_text_parts += [text_part]
+                    continue
+                raise RuntimeError(
+                    f"An exception occurred while loading multimodal data: {e}"
+                )
             except Exception as e:
                 raise RuntimeError(
                     f"An exception occurred while loading multimodal data: {e}"
@@ -601,7 +692,6 @@ class BaseMultimodalProcessor(ABC):
         mask = input_ids == mm_token_id
         start_positions = (mask & ~torch.roll(mask, 1)).nonzero(as_tuple=True)[0]
         end_positions = (mask & ~torch.roll(mask, -1)).nonzero(as_tuple=True)[0]
-
         return list(zip(start_positions.tolist(), end_positions.tolist()))
 
     @staticmethod
@@ -614,35 +704,42 @@ class BaseMultimodalProcessor(ABC):
         return list(zip(indices_start.tolist(), indices_end.tolist()))
 
     def collect_mm_items_from_processor_output(
-        self, data_dict: dict
+        self, data_dict: dict, modality: Modality = None
     ) -> List[MultimodalDataItem]:
-        """Create mm_items directly from processor output."""
+        """
+        Create mm_items directly from processor output, with one item for each modality
+
+        Note that the data_dict can be passed via offline engine api
+        """
+
         items: dict[Modality, MultimodalDataItem] = {}
         for attr_name, value in data_dict.items():
             if attr_name == "input_ids":
                 continue
 
             # Get modality for this attribute
-            modality = self.ATTR_NAME_TO_MODALITY.get(attr_name)
+            current_modality = modality or self.ATTR_NAME_TO_MODALITY.get(attr_name)
 
             if attr_name == "precomputed_embeddings":
                 modality_str = data_dict.get("modality")
-                modality = Modality.IMAGE
+                current_modality = Modality.IMAGE
                 if modality_str:
                     try:
-                        modality = Modality.from_str(modality_str)
+                        current_modality = Modality.from_str(modality_str)
                     except ValueError:
                         pass
 
-            if modality:
+            if current_modality:
                 # Create item if needed
-                if modality not in items:
+                if current_modality not in items:
-                    items[modality] = MultimodalDataItem(modality=modality)
+                    items[current_modality] = MultimodalDataItem(
+                        modality=current_modality
+                    )
 
                 if attr_name in self.FEATURE_NAMES:
                     attr_name = "feature"
 
-                items[modality].set(attr_name, value)
+                items[current_modality].set(attr_name, value)
 
         return list(items.values())
 
@@ -678,9 +775,9 @@ class BaseMultimodalProcessor(ABC):
             Tuple of (list of mm_items, input_ids)
         """
         # Collect all items and categorize them
-        all_items = base_output.organize_results()
+        all_loaded_data = base_output.organize_results()
         # Handle text-only case
-        if not all_items:
+        if not all_loaded_data:
             input_ids = self._processor.tokenizer(
                 base_output.input_text,
                 return_tensors="pt",
@@ -689,9 +786,9 @@ class BaseMultimodalProcessor(ABC):
             return [], input_ids, {}
 
         dict_items, raw_images, raw_audios, raw_videos = [], [], [], []
-        for modality, item in all_items:
+        for modality, item in all_loaded_data:
             if isinstance(item, dict):
-                dict_items.append(item)
+                dict_items.append((modality, item))
             elif modality == Modality.IMAGE:
                 raw_images.append(item)
             elif modality == Modality.AUDIO:
@@ -717,12 +814,25 @@ class BaseMultimodalProcessor(ABC):
         else:
             ret = None
 
-        # Handle dict items (already processed)
+        # Handle dict items (processed or precomputed)
-        for dict_item in dict_items:
+        for modality, dict_item in dict_items:
-            all_collected_items.extend(
+            input_format = dict_item.get("format", None)
-                self.collect_mm_items_from_processor_output(dict_item)
+            if input_format == "processor_output":
+                items = self.collect_mm_items_from_processor_output(dict_item)
+                for item in items:
+                    item.format = MultimodalInputFormat.PROCESSOR_OUTPUT
+                all_collected_items.extend(items)
+            elif input_format == "precomputed_embedding":
+                feature = dict_item["feature"]
+                del dict_item["feature"]
+                all_collected_items.append(
+                    MultimodalDataItem(
+                        modality=modality,
+                        feature=feature,
+                        format=MultimodalInputFormat.PRECOMPUTED_EMBEDDING,
+                        model_specific_data=dict_item,
+                    )
                 )
-
         # Fallback tokenization if no raw items were processed
         if input_ids is None:
             input_ids = self._processor.tokenizer(

python/sglang/srt/multimodal/processors/llava.py

@@ -1,5 +1,5 @@
 import asyncio
-from typing import List, Optional, Union
+from typing import Dict, List, Optional, Union
 
 import numpy as np
 from transformers.models.auto.processing_auto import (
@@ -106,6 +106,32 @@ class LlavaImageProcessor(BaseMultimodalProcessor):
                 self._processor.image_processor,
             )
 
+    def _process_precomputed_image_data(self, image_data: List[Dict]) -> Dict:
+        mm_items = []
+        for item in image_data:
+            # Infer size logic...
+            if "image_sizes" not in item:
+                if "pixel_values" in item:
+                    pv = item["pixel_values"]
+                    # Handle simplified if/else
+                    h, w = (
+                        (pv.shape[2], pv.shape[3])
+                        if len(pv.shape) == 4
+                        else (pv.shape[1], pv.shape[2])
+                    )
+                    item["image_sizes"] = [(w, h)]
+                else:
+                    item["image_sizes"] = [(336, 336)]
+
+            mm_items.append(
+                MultimodalDataItem(
+                    feature=item["feature"],
+                    modality=Modality.IMAGE,
+                    model_specific_data=item,
+                )
+            )
+        return {"mm_items": mm_items}
+
     async def process_mm_data_async(
         self,
         image_data: List[Union[str, bytes, ImageData]],
@@ -114,6 +140,17 @@ class LlavaImageProcessor(BaseMultimodalProcessor):
         *args,
         **kwargs,
     ):
+        # FIX: Handle precomputed embeddings (dictionaries)
+        # If the input is already a dictionary, we skip the CPU image processor.
+        # We also need to infer 'image_sizes' from 'pixel_values' if missing,
+        # because pad_input_ids requires it.
+        if (
+            isinstance(image_data, list)
+            and len(image_data) > 0
+            and isinstance(image_data[0], dict)
+        ):
+            return self._process_precomputed_image_data(image_data)
+
         modalities = request_obj.modalities or ["image"]
         aspect_ratio = getattr(self.hf_config, "image_aspect_ratio", None)
         grid_pinpoints = (
@@ -180,6 +217,8 @@ class LlavaMultimodalProcessor(BaseMultimodalProcessor):
     models = [LlavaForConditionalGeneration, Mistral3ForConditionalGeneration]
 
     def _get_sgl_processor_cls(self, model_type: str):
+        if model_type == "clip_vision_model":
+            return LlavaImageProcessor
         if hf_name := HF_MAPPING_NAMES.get(model_type):
             sgl_mm_processor_set = sgl_mm_processor_utils.PROCESSOR_MAPPING.values()
             sgl_processor_cls = list(

python/sglang/srt/multimodal/processors/qwen_vl.py

@@ -347,14 +347,30 @@ class QwenVLImageProcessor(SGLangBaseProcessor):
                     audio_item.feature_attention_mask, dim=1
                 )
 
-        second_per_grid_ts = getattr(ret, "second_per_grid_ts", None) or getattr(
+        second_per_grid_ts = getattr(ret, "second_per_grid_ts", None)
-            ret, "video_second_per_grid", None
+        if second_per_grid_ts is None:
-        )
+            second_per_grid_ts = getattr(ret, "video_second_per_grid", None)
 
         process_time = time.perf_counter()
 
         input_ids = input_ids.flatten()
 
+        image_grid_thw = None
+        if hasattr(ret, "image_grid_thw"):
+            image_grid_thw = ret.image_grid_thw
+
+        if image_grid_thw is None and image_data and isinstance(image_data[0], dict):
+            image_grid_thw = image_data[0].get("image_grid_thw")
+
+        video_grid_thw = None
+        if hasattr(ret, "video_grid_thw"):
+            video_grid_thw = ret.video_grid_thw
+
+        if video_grid_thw is None and request_obj.video_data:
+            first_video = request_obj.video_data[0]
+            if isinstance(first_video, dict):
+                video_grid_thw = first_video.get("video_grid_thw")
+
         mrope_positions, mrope_position_delta = MRotaryEmbedding.get_rope_index(
             spatial_merge_size=self.hf_config.vision_config.spatial_merge_size,
             image_token_id=self.mm_tokens.image_token_id,
@@ -364,6 +380,7 @@ class QwenVLImageProcessor(SGLangBaseProcessor):
             tokens_per_second=getattr(
                 self.hf_config.vision_config, "tokens_per_second", None
             ),
+            # use the expanded token ids
             input_ids=input_ids.unsqueeze(0),
             image_grid_thw=getattr(ret, "image_grid_thw", None),
             video_grid_thw=getattr(ret, "video_grid_thw", None),

python/sglang/test/run_eval.py

@@ -39,7 +39,7 @@ def run_eval_once(args, base_url: str, eval_obj: Eval) -> dict:
         base_url=base_url,
         temperature=getattr(args, "temperature", 0.0),
         reasoning_effort=getattr(args, "reasoning_effort", None),
-        extra_body=thinking_kwargs,
+        extra_body=thinking_kwargs if thinking_kwargs else None,
     )
 
     # Run eval

test/srt/test_vision_openai_server_a.py

@@ -219,12 +219,15 @@ class TestDeepseekOCRServer(TestOpenAIMLLMServerBase):
         self.verify_single_image_response_for_ocr(response)
 
 
-if __name__ == "__main__":
+# Delete the mixin classes so that they are not collected by pytest
-    del (
+del (
     TestOpenAIMLLMServerBase,
     ImageOpenAITestMixin,
     VideoOpenAITestMixin,
     AudioOpenAITestMixin,
     OmniOpenAITestMixin,
-    )
+)
+
+
+if __name__ == "__main__":
     unittest.main()

test/srt/test_vlm_input_format.py

@@ -1,20 +1,41 @@
 import json
 import unittest
+from io import BytesIO
 from typing import Optional
 
+import requests
 import torch
+
+# Compatibility shim: Kimi-VL dynamic module expects PytorchGELUTanh which may
+# be missing in transformers==4.57.1. Inject a lightweight implementation so
+# the model can import successfully without downgrading transformers.
+import transformers.activations as _hf_activations
+from PIL import Image
 from transformers import (
+    AutoModel,
     AutoProcessor,
     Gemma3ForConditionalGeneration,
     Qwen2_5_VLForConditionalGeneration,
 )
 
+if not hasattr(_hf_activations, "PytorchGELUTanh"):
+
+    class PytorchGELUTanh(torch.nn.Module):
+        def forward(self, x):
+            return torch.nn.functional.gelu(x, approximate="tanh")
+
+    _hf_activations.PytorchGELUTanh = PytorchGELUTanh
+    _hf_activations.ACT2FN.setdefault(
+        "pytorch_gelu_tanh",
+        lambda x: torch.nn.functional.gelu(x, approximate="tanh"),
+    )
+
 from sglang import Engine
 from sglang.srt.entrypoints.openai.protocol import ChatCompletionRequest
 from sglang.srt.parser.conversation import generate_chat_conv
-from sglang.test.test_utils import download_image_with_retry
 
-TEST_IMAGE_URL = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
+IMAGE_MAN_IRONING_URL = "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/images/man_ironing_on_back_of_suv.png"
+IMAGE_SGL_LOGO_URL = "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/images/sgl_logo.png"
 
 
 class VLMInputTestBase:
@@ -27,9 +48,12 @@ class VLMInputTestBase:
     def setUpClass(cls):
         assert cls.model_path is not None, "Set model_path in subclass"
         assert cls.chat_template is not None, "Set chat_template in subclass"
-        cls.image_url = TEST_IMAGE_URL
+        cls.image_urls = [IMAGE_MAN_IRONING_URL, IMAGE_SGL_LOGO_URL]
         cls.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-        cls.main_image = download_image_with_retry(cls.image_url)
+        cls.main_image = []
+        for image_url in cls.image_urls:
+            response = requests.get(image_url)
+            cls.main_image.append(Image.open(BytesIO(response.content)))
         cls.processor = AutoProcessor.from_pretrained(
             cls.model_path, trust_remote_code=True, use_fast=True
         )
@@ -55,8 +79,30 @@ class VLMInputTestBase:
         self.engine.shutdown()
 
     def verify_response(self, output):
+        # The goal is to check that the model roughly understands:
+        #   - image 1: taxi / car scene
+        #   - image 2: SGL logo / company
+        # We intentionally keep the check keyword-based and loose to avoid
+        # overfitting to a specific phrasing.
         out_text = output["text"].lower()
-        assert "taxi" in out_text or "cab" in out_text or "car" in out_text, out_text
+
+        assert any(w in out_text for w in ("taxi", "cab", "car")), out_text
+
+        has_sg_or_logo_side = any(
+            kw in out_text
+            for kw in (
+                "sg ",
+                "sgl",
+                " sgl",
+                "logo",
+                "software guidance",
+                "labs",
+                "laborator",
+                "company",
+                " text",
+            )
+        )
+        assert has_sg_or_logo_side, out_text
 
     def get_completion_request(self) -> ChatCompletionRequest:
         json_structure = {
@@ -65,8 +111,12 @@ class VLMInputTestBase:
                 {
                     "role": "user",
                     "content": [
-                        {"type": "image_url", "image_url": {"url": self.image_url}},
+                        {"type": "image_url", "image_url": {"url": self.image_urls[0]}},
-                        {"type": "text", "text": "What's in this picture?"},
+                        {"type": "image_url", "image_url": {"url": self.image_urls[1]}},
+                        {
+                            "type": "text",
+                            "text": "Describe both the first image and the second image in detail separately.",  # update prompt, ensure kimi-vl understands the images separately.
+                        },
                     ],
                 }
             ],
@@ -83,55 +133,58 @@ class VLMInputTestBase:
         # Process inputs using processor
         inputs = self.processor(
             text=[text],
-            images=[self.main_image],
+            images=self.main_image,
             return_tensors="pt",
         ).to(self.device)
 
-        return inputs
+        return inputs, text
 
-    async def test_understands_image(self):
+    async def test_accepts_image(self):
         req = self.get_completion_request()
         conv = generate_chat_conv(req, template_name=self.chat_template)
         text = conv.get_prompt()
         output = await self.engine.async_generate(
             prompt=text,
-            image_data=[self.main_image],
+            image_data=self.main_image,
-            sampling_params=dict(temperature=0.0),
+            sampling_params=dict(temperature=0.0, max_new_tokens=512),
         )
         self.verify_response(output)
 
-    async def test_understands_precomputed_embeddings(self):
+    async def test_accepts_precomputed_embeddings(self):
         req = self.get_completion_request()
-        processor_output = self.get_processor_output(req=req)
+        processor_output, _ = self.get_processor_output(req=req)
+
         with torch.inference_mode():
             precomputed_embeddings = self.__class__.visual(processor_output)
+
         output = await self.engine.async_generate(
             input_ids=processor_output["input_ids"][0].detach().cpu().tolist(),
             image_data=[
                 self._precomputed_image_data(processor_output, precomputed_embeddings)
             ],
-            sampling_params=dict(temperature=0.0),
+            sampling_params=dict(temperature=0.0, max_new_tokens=512),
         )
         self.verify_response(output)
 
-    async def test_understands_pixel_values(self):
+    async def test_accepts_processor_output(self):
         req = self.get_completion_request()
-        processor_output = self.get_processor_output(req=req)
+        processor_output, prompt = self.get_processor_output(req=req)
         output = await self.engine.async_generate(
             input_ids=processor_output["input_ids"][0].detach().cpu().tolist(),
-            image_data=[self._pixel_values_image_data(processor_output)],
+            image_data=[self._processor_output_image_data(processor_output)],
-            sampling_params=dict(temperature=0.0),
+            sampling_params=dict(temperature=0.0, max_new_tokens=512),
         )
         self.verify_response(output)
 
     def _precomputed_image_data(self, processor_output, precomputed_embeddings):
         """This should not be overridden."""
         return dict(
-            modality="IMAGE",
+            processor_output,
-            precomputed_embeddings=precomputed_embeddings,
+            format="precomputed_embedding",
+            feature=precomputed_embeddings,
         )
 
-    def _pixel_values_image_data(self, processor_output):
+    def _processor_output_image_data(self, processor_output):
         """Override in subclass to pass the correct set of arguments."""
         raise NotImplementedError
 
@@ -153,12 +206,8 @@ class TestQwenVLUnderstandsImage(VLMInputTestBase, unittest.IsolatedAsyncioTestC
             processor_output["pixel_values"], processor_output["image_grid_thw"]
         )
 
-    def _pixel_values_image_data(self, processor_output):
+    def _processor_output_image_data(self, processor_output):
-        return dict(
+        return dict(processor_output, format="processor_output")
-            modality="IMAGE",
-            image_grid_thw=processor_output["image_grid_thw"],
-            pixel_values=processor_output["pixel_values"],
-        )
 
 
 class TestGemmaUnderstandsImage(VLMInputTestBase, unittest.IsolatedAsyncioTestCase):
@@ -170,57 +219,60 @@ class TestGemmaUnderstandsImage(VLMInputTestBase, unittest.IsolatedAsyncioTestCa
         model = Gemma3ForConditionalGeneration.from_pretrained(
             cls.model_path, torch_dtype=torch.bfloat16
         )
-        cls.vision_tower = model.vision_tower.eval().to(cls.device)
+        base_model = model.model
+
+        cls.vision_tower = base_model.vision_tower.eval().to(cls.device)
+
+        if hasattr(base_model, "multi_modal_projector"):
+            cls.mm_projector = base_model.multi_modal_projector.eval().to(cls.device)
+        else:
             cls.mm_projector = model.multi_modal_projector.eval().to(cls.device)
+
         cls.visual = lambda processor_output: cls.mm_projector(
             cls.vision_tower(
                 pixel_values=processor_output["pixel_values"]
             ).last_hidden_state
         )
 
-    def _pixel_values_image_data(self, processor_output):
+    def _processor_output_image_data(self, processor_output):
-        return dict(
+        return dict(processor_output, format="processor_output")
-            modality="IMAGE",
+
-            pixel_values=processor_output["pixel_values"][0],
+
+# Updated Kimi-VL test to use the new input format.
+class TestKimiVLImageUnderstandsImage(
+    VLMInputTestBase, unittest.IsolatedAsyncioTestCase
+):
+    model_path = "moonshotai/Kimi-VL-A3B-Instruct"
+    chat_template = "kimi-vl"
+
+    @classmethod
+    def _init_visual(cls):
+        model = AutoModel.from_pretrained(cls.model_path, trust_remote_code=True)
+        cls.vision_tower = model.vision_tower.eval().to(cls.device)
+        cls.mm_projector = model.multi_modal_projector.eval().to(cls.device)
+
+        cls.visual = lambda tokenizer_output: cls.mm_projector(
+            cls.vision_tower(
+                pixel_values=tokenizer_output["pixel_values"],
+                grid_hws=tokenizer_output["image_grid_hws"],
+            )
         )
 
-
+    def _processor_output_image_data(self, processor_output):
-# Temporarily skip Kimi-VL for CI test due to issue in transformers=4.57.0
+        return dict(processor_output, format="processor_output")
-# class TestKimiVLImageUnderstandsImage(
-#     VLMInputTestBase, unittest.IsolatedAsyncioTestCase
-# ):
-#     model_path = "moonshotai/Kimi-VL-A3B-Instruct"
-#     chat_template = "kimi-vl"
-
-#     @classmethod
-#     def _init_visual(cls):
-#         model = AutoModel.from_pretrained(cls.model_path, trust_remote_code=True)
-#         cls.vision_tower = model.vision_tower.eval().to(cls.device)
-#         cls.mm_projector = model.multi_modal_projector.eval().to(cls.device)
-
-#         cls.visual = lambda tokenizer_output: cls.mm_projector(
-#             cls.vision_tower(
-#                 pixel_values=tokenizer_output["pixel_values"],
-#                 grid_hws=tokenizer_output["image_grid_hws"],
-#             )
-#         )
-
-#     def _pixel_values_image_data(self, processor_output):
-#         return dict(
-#             modality="IMAGE",
-#             pixel_values=processor_output["pixel_values"],
-#             image_grid_hws=processor_output["image_grid_hws"],
-#         )
 
 
 # not for CI: too large
 # class TestLlama4ImageUnderstandsImage(
 #     VLMInputTestBase, unittest.IsolatedAsyncioTestCase
 # ):
+#     # Allow overriding via env for local/offline runs.
 #     model_path = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
-#     chat_template = "llama_4_vision"
+#     chat_template = "llama-4"
 
 #     def setUp(self):
+#         if torch.cuda.device_count() < 4:
+#             self.skipTest("Skipping Llama-4 test: requires 4 GPUs for TP=4")
 #         self.engine = Engine(
 #             model_path=self.model_path,
 #             trust_remote_code=True,
@@ -234,7 +286,12 @@ class TestGemmaUnderstandsImage(VLMInputTestBase, unittest.IsolatedAsyncioTestCa
 
 #     @classmethod
 #     def _init_visual(cls):
-#         model = AutoModel.from_pretrained(cls.model_path, trust_remote_code=True, torch_dtype="auto")
+#         model = AutoModel.from_pretrained(
+#             cls.model_path,
+#             trust_remote_code=True,
+#             torch_dtype="auto",
+#             force_download=True,
+#         )
 #         cls.vision_tower = model.vision_model.eval().to(cls.device)
 #         cls.mm_projector = model.multi_modal_projector.eval().to(cls.device)
 
@@ -244,11 +301,48 @@ class TestGemmaUnderstandsImage(VLMInputTestBase, unittest.IsolatedAsyncioTestCa
 #             ).last_hidden_state.flatten(0, -2)
 #         )
 
-#     def _pixel_values_image_data(self, processor_output):
+#     def _processor_output_image_data(self, processor_output):
-#         return dict(
+#         # Llama-4 vision expects processor_output format with pixel_values
-#             modality="IMAGE",
+#         return dict(processor_output, format="processor_output")
-#             pixel_values=processor_output["pixel_values"],
+
+
+# class TestLlavaUnderstandsImage(VLMInputTestBase, unittest.IsolatedAsyncioTestCase):
+#     model_path = "llava-hf/llava-1.5-7b-hf"
+#     chat_template = "vicuna_v1.1"
+
+#     @classmethod
+#     def _init_visual(cls):
+#         from transformers import LlavaForConditionalGeneration
+
+#         model = LlavaForConditionalGeneration.from_pretrained(
+#             cls.model_path,
+#             torch_dtype=torch.float16,
+#             low_cpu_mem_usage=True,
 #         )
+#         cls.vision_tower = model.vision_tower.eval().to(cls.device)
+#         cls.multi_modal_projector = model.multi_modal_projector.eval().to(cls.device)
+#         cls.config = model.config
+
+#         def visual_func(processor_output):
+#             pixel_values = processor_output["pixel_values"].to(
+#                 cls.device, dtype=torch.float16
+#             )
+
+#             vision_outputs = cls.vision_tower(pixel_values, output_hidden_states=True)
+#             image_features = vision_outputs.hidden_states[-2]
+
+#             if cls.config.vision_feature_select_strategy == "default":
+#                 image_features = image_features[:, 1:]
+#             elif cls.config.vision_feature_select_strategy == "full":
+#                 image_features = image_features
+
+#             image_features = cls.multi_modal_projector(image_features)
+#             return image_features
+
+#         cls.visual = visual_func
+
+#     def _processor_output_image_data(self, processor_output):
+#         return dict(processor_output, format="processor_output")
 
 
 if __name__ == "__main__":