model: support baidu unlimited-ocr (#29186)
Co-authored-by: Mick <mickjagger19@icloud.com>
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
<svg width="940" height="525" viewBox="0 0 940 525" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<rect width="940" height="525" fill="none"/>
|
||||
<text x="470" y="286" text-anchor="middle" font-family="Arial, Helvetica, sans-serif" font-size="128" font-weight="700" letter-spacing="0">
|
||||
<tspan fill="#2B5BFF">Bai</tspan><tspan fill="#D5001C">du</tspan>
|
||||
</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 376 B |
@@ -0,0 +1,155 @@
|
||||
---
|
||||
title: Unlimited-OCR
|
||||
description: "Deploy Baidu Unlimited-OCR with SGLang for long document OCR using prefill-aware sliding-window attention."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Unlimited-OCR support is in [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186). Until that PR is included in a tagged SGLang release, install from a build that contains the PR.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git fetch origin pull/29186/head && git checkout FETCH_HEAD
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware to generate the launch command. The recipe uses FlashAttention-3 with `--page-size 1`, which is required by the current prefill-aware sliding-window attention path. It also disables radix cache by default, which is the better fit for batch OCR workloads where each request usually contains a different image.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/baidu/unlimited-ocr.jsx";
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
## Playground
|
||||
|
||||
Use the Playground to adjust tensor parallelism on top of the selected deployment cell.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is Baidu's multimodal OCR model for document parsing. It uses a sliding-window language backbone, but SGLang serves it with a prefill-aware sliding-window path so image and prompt tokens remain visible during long decode.
|
||||
|
||||
The SGLang integration loads the standalone Unlimited-OCR architecture with SAM and CLIP vision encoders plus a DeepSeek-style language backbone. It supports OpenAI-compatible image requests and model-specific image processing options through `images_config`.
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/baidu/Unlimited-OCR) · [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Attention backend**: use `--attention-backend fa3 --page-size 1`. The prefill-aware SWA page table is built with token-level locations, so page size 1 is required.
|
||||
- **Radix cache**: keep `--disable-radix-cache` for batch OCR over different documents. If your workload repeatedly asks about the same image and prompt, remove this flag to allow prefix reuse through `PureSWARadixCache`.
|
||||
- **Long OCR generations**: keep the default prefill-aware SWA path enabled. It retains prompt and image KV while still applying a sliding window to generated text.
|
||||
- **Custom logit processor**: keep `--enable-custom-logit-processor` in the launch command.
|
||||
- **Image modes**: pass `images_config.image_mode` per request. Supported modes are `tiny`, `small`, `base`, `large`, and `gundam`. Multiple images are supported only for `tiny`, `small`, and `base`.
|
||||
- **Default image mode**: when `images_config.image_mode` is omitted, SGLang uses `gundam`.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 OCR request
|
||||
|
||||
<Accordion title="OCR Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="baidu/Unlimited-OCR",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "document parsing."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://example.com/your_document.png"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=2048,
|
||||
temperature=0,
|
||||
extra_body={"images_config": {"image_mode": "gundam"}},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Choosing an image mode
|
||||
|
||||
Use lower modes to reduce prefill cost for simple images, and use `gundam` for high-detail document parsing.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Use</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Multiple images</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>tiny</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Lowest prefill cost.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>small</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Lightweight OCR requests.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>base</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Balanced quality and cost.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>large</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Higher resolution single-image OCR.</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>gundam</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Default high-detail document parsing mode.</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -67,6 +67,12 @@ metatags:
|
||||
href="/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra"
|
||||
img="/cards/logos/nvidia.png"
|
||||
/>
|
||||
<Card
|
||||
title="Baidu"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Baidu/Unlimited-OCR"
|
||||
img="/cards/logos/baidu.svg"
|
||||
/>
|
||||
<Card
|
||||
title="Ernie"
|
||||
mode="card"
|
||||
|
||||
@@ -201,6 +201,10 @@
|
||||
"source": "/advanced_features/vlm_query.html",
|
||||
"destination": "/docs/advanced_features/vlm_query"
|
||||
},
|
||||
{
|
||||
"source": "/basic_usage/unlimited_ocr.html",
|
||||
"destination": "/cookbook/autoregressive/Baidu/Unlimited-OCR"
|
||||
},
|
||||
{
|
||||
"source": "/basic_usage/deepseek_ocr.html",
|
||||
"destination": "/cookbook/autoregressive/DeepSeek/DeepSeek-OCR"
|
||||
@@ -1058,6 +1062,12 @@
|
||||
"cookbook/autoregressive/NVIDIA/Nemotron3-Super"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Baidu",
|
||||
"pages": [
|
||||
"cookbook/autoregressive/Baidu/Unlimited-OCR"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "Ernie",
|
||||
"pages": [
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
// Unlimited-OCR cookbook config. Consumed by _deployment.jsx + _playground.jsx.
|
||||
|
||||
export const config = {
|
||||
modelName: "Unlimited-OCR",
|
||||
|
||||
supportedHardware: ["h100", "h200", "b200", "b300", "gb200", "gb300"],
|
||||
|
||||
variants: [{ id: "default", label: "Default" }],
|
||||
quantizations: [{ id: "default", label: "Default" }],
|
||||
strategies: [{ id: "balanced", label: "Balanced" }],
|
||||
nodesOptions: [{ id: "single", label: "Single Node" }],
|
||||
|
||||
modelNames: {
|
||||
"default|default": "baidu/Unlimited-OCR",
|
||||
},
|
||||
|
||||
placeholders: {
|
||||
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||
HF_TOKEN: {
|
||||
target: "command",
|
||||
label: "HF token (Docker)",
|
||||
default: "<your-hf-token>",
|
||||
},
|
||||
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||
},
|
||||
|
||||
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
|
||||
-H 'Content-Type: application/json' \\
|
||||
-d '{
|
||||
"model": "{{MODEL_NAME}}",
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "document parsing."},
|
||||
{"type": "image_url", "image_url": {"url": "https://example.com/your_document.png"}}
|
||||
]
|
||||
}],
|
||||
"images_config": {"image_mode": "gundam"},
|
||||
"temperature": 0,
|
||||
"max_tokens": 2048
|
||||
}'`,
|
||||
|
||||
dockerImages: {
|
||||
h100: "lmsysorg/sglang:dev",
|
||||
h200: "lmsysorg/sglang:dev",
|
||||
b200: "lmsysorg/sglang:dev",
|
||||
b300: "lmsysorg/sglang:dev",
|
||||
gb200: "lmsysorg/sglang:dev",
|
||||
gb300: "lmsysorg/sglang:dev",
|
||||
},
|
||||
|
||||
github: {
|
||||
cookbookModel: "baidu/Unlimited-OCR",
|
||||
},
|
||||
|
||||
playgroundFeatures: {
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [null, 1, 2, 4, 8] },
|
||||
],
|
||||
},
|
||||
},
|
||||
|
||||
cells: [
|
||||
{
|
||||
match: {
|
||||
hw: "h100",
|
||||
variant: "default",
|
||||
quant: "default",
|
||||
strategy: "balanced",
|
||||
nodes: "single",
|
||||
},
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--attention-backend fa3",
|
||||
"--page-size 1",
|
||||
"--context-length 32768",
|
||||
"--enable-custom-logit-processor",
|
||||
"--disable-radix-cache",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: {
|
||||
hw: "h200",
|
||||
variant: "default",
|
||||
quant: "default",
|
||||
strategy: "balanced",
|
||||
nodes: "single",
|
||||
},
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--attention-backend fa3",
|
||||
"--page-size 1",
|
||||
"--context-length 32768",
|
||||
"--enable-custom-logit-processor",
|
||||
"--disable-radix-cache",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: {
|
||||
hw: "b200",
|
||||
variant: "default",
|
||||
quant: "default",
|
||||
strategy: "balanced",
|
||||
nodes: "single",
|
||||
},
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--attention-backend fa3",
|
||||
"--page-size 1",
|
||||
"--context-length 32768",
|
||||
"--enable-custom-logit-processor",
|
||||
"--disable-radix-cache",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: {
|
||||
hw: "b300",
|
||||
variant: "default",
|
||||
quant: "default",
|
||||
strategy: "balanced",
|
||||
nodes: "single",
|
||||
},
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--attention-backend fa3",
|
||||
"--page-size 1",
|
||||
"--context-length 32768",
|
||||
"--enable-custom-logit-processor",
|
||||
"--disable-radix-cache",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: {
|
||||
hw: "gb200",
|
||||
variant: "default",
|
||||
quant: "default",
|
||||
strategy: "balanced",
|
||||
nodes: "single",
|
||||
},
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--attention-backend fa3",
|
||||
"--page-size 1",
|
||||
"--context-length 32768",
|
||||
"--enable-custom-logit-processor",
|
||||
"--disable-radix-cache",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: {
|
||||
hw: "gb300",
|
||||
variant: "default",
|
||||
quant: "default",
|
||||
strategy: "balanced",
|
||||
nodes: "single",
|
||||
},
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--attention-backend fa3",
|
||||
"--page-size 1",
|
||||
"--context-length 32768",
|
||||
"--enable-custom-logit-processor",
|
||||
"--disable-radix-cache",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
Reference in New Issue
Block a user