190 lines
8.9 KiB
Plaintext
190 lines
8.9 KiB
Plaintext
---
|
||
title: Embedding models
|
||
description: Dense and sparse embedding models with FlashInfer acceleration and SGLang's batching infrastructure.
|
||
---
|
||
SGLang provides robust support for embedding models by integrating efficient serving mechanisms with its flexible programming interface. This integration allows for streamlined handling of embedding tasks, facilitating faster and more accurate retrieval and semantic search operations. SGLang's architecture enables better resource utilization and reduced latency in embedding model deployment.
|
||
|
||
<Warning>
|
||
Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models require `--is-embedding`; add `--trust-remote-code` when the model requires it.
|
||
</Warning>
|
||
|
||
## Quick Start
|
||
|
||
### Launch Server
|
||
|
||
```bash
|
||
sglang serve \
|
||
--model-path Qwen/Qwen3-Embedding-4B \
|
||
--is-embedding
|
||
```
|
||
|
||
### EmbeddingGemma
|
||
|
||
EmbeddingGemma uses bidirectional attention and is auto-detected, so its best default command is simply:
|
||
|
||
```bash
|
||
sglang serve --model-path google/embeddinggemma-300m
|
||
```
|
||
|
||
On CUDA, SGLang automatically uses breakable CUDA graph (BCG) for its full encoder prefill and disables incompatible radix-cache and chunked-prefill behavior. Do not add the deprecated piecewise CUDA graph knobs.
|
||
|
||
### Client Request
|
||
|
||
```python
|
||
import requests
|
||
|
||
url = "http://127.0.0.1:30000"
|
||
|
||
payload = {
|
||
"model": "Qwen/Qwen3-Embedding-4B",
|
||
"input": "What is the capital of France?",
|
||
"encoding_format": "float" # or "base64" for compact FP32 responses
|
||
}
|
||
|
||
response = requests.post(url + "/v1/embeddings", json=payload).json()
|
||
print("Embedding:", response["data"][0]["embedding"])
|
||
```
|
||
|
||
|
||
## Multimodal Embedding Example
|
||
|
||
For multimodal models like GME that support both text and images:
|
||
|
||
```bash
|
||
sglang serve \
|
||
--model-path Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
|
||
--is-embedding \
|
||
--chat-template gme-qwen2-vl
|
||
```
|
||
|
||
```python Example
|
||
import requests
|
||
|
||
url = "http://127.0.0.1:30000"
|
||
|
||
text_input = "Represent this image in embedding space."
|
||
image_path = "https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild/resolve/main/images/023.jpg"
|
||
|
||
payload = {
|
||
"model": "gme-qwen2-vl",
|
||
"input": [
|
||
{
|
||
"text": text_input
|
||
},
|
||
{
|
||
"image": image_path
|
||
}
|
||
],
|
||
}
|
||
|
||
response = requests.post(url + "/v1/embeddings", json=payload).json()
|
||
|
||
print("Embeddings:", [x.get("embedding") for x in response.get("data", [])])
|
||
```
|
||
|
||
## Matryoshka Embedding Example
|
||
|
||
[Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows user to trade off between performance and cost.
|
||
|
||
### 1. Launch a Matryoshka‑capable model
|
||
|
||
If the model config already includes `matryoshka_dimensions` or `is_matryoshka` then no override is needed. Otherwise, you can use `--json-model-override-args` as below:
|
||
|
||
```bash Command
|
||
sglang serve \
|
||
--model-path Qwen/Qwen3-Embedding-0.6B \
|
||
--is-embedding \
|
||
--json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 1024, 1536]}'
|
||
```
|
||
|
||
1. Setting `"is_matryoshka": true` allows truncating to any dimension. Otherwise, the server will validate that the specified dimension in the request is one of `matryoshka_dimensions`.
|
||
2. Omitting `dimensions` in a request returns the full vector.
|
||
|
||
### 2. Make requests with different output dimensions
|
||
|
||
```python
|
||
import requests
|
||
|
||
url = "http://127.0.0.1:30000"
|
||
|
||
# Request a truncated (Matryoshka) embedding by specifying a supported dimension.
|
||
payload = {
|
||
"model": "Qwen/Qwen3-Embedding-0.6B",
|
||
"input": "Explain diffusion models simply.",
|
||
"dimensions": 512 # change to 128 / 1024 / omit for full size
|
||
}
|
||
|
||
response = requests.post(url + "/v1/embeddings", json=payload).json()
|
||
print("Embedding:", response["data"][0]["embedding"])
|
||
```
|
||
|
||
|
||
## Supported Models
|
||
|
||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||
<colgroup>
|
||
<col style={{width: "25%"}} />
|
||
<col style={{width: "25%"}} />
|
||
<col style={{width: "25%"}} />
|
||
<col style={{width: "25%"}} />
|
||
</colgroup>
|
||
<thead>
|
||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Family</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Example Model</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Chat template</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>EmbeddingGemma</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`google/embeddinggemma-300m`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Bidirectional Gemma3 text encoder; auto-detected and served with BCG by default</td>
|
||
</tr>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>E5 (Llama/Mistral based)</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`intfloat/e5-mistral-7b-instruct`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>High-quality text embeddings based on Mistral/Llama architectures</td>
|
||
</tr>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GTE-Qwen2</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Alibaba-NLP/gte-Qwen2-7B-instruct`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Alibaba's text embedding model with multilingual support</td>
|
||
</tr>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-Embedding</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Qwen/Qwen3-Embedding-4B`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Latest Qwen3-based text embedding model for semantic representation</td>
|
||
</tr>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3 (bare backbone)</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`microsoft/harrier-oss-v1-0.6b`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Bare <code>Qwen3Model</code> backbone (no LM head); served natively on SGLang's fused Qwen3 kernels and auto-classified as an embedding model</td>
|
||
</tr>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>BGE</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`BAAI/bge-large-en-v1.5`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BAAI's text embeddings (requires <code>attention-backend</code> triton/torch_native)</td>
|
||
</tr>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GME (Multimodal)</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Alibaba-NLP/gme-Qwen2-VL-2B-Instruct`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`gme-qwen2-vl`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Multimodal embedding for text and image cross-modal tasks</td>
|
||
</tr>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CLIP</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`openai/clip-vit-large-patch14-336`</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>OpenAI's CLIP for image and text embeddings</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|