minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

embedding: centralize capabilities and complete OpenAI compatibility (#32481)

Browse Source
This commit is contained in:
Mick
2026-07-30 10:28:52 +08:00
committed by GitHub
parent 313a518bee
commit 22faf9fef8
16 changed files with 728 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/basic_usage/native_api.mdx
+2
View File
@@ -64,6 +64,7 @@ Get the information of the model.
- `has_audio_understanding`: Whether the model has audio-understanding capability.
- `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama").
- `architectures`: The model architectures from the HuggingFace config (e.g., ["Qwen2ForCausalLM"]).
- `embedding`: The resolved embedding-serving plan. It includes pooling, normalization, execution and attention style, Matryoshka dimensions, cache policy, and effective BCG prefill settings. This field is available when the model configuration exposes an embedding capability contract.
```python Example
url = f"http://localhost:{port}/get_model_info"
@@ -85,6 +86,7 @@ assert response_json.keys() == {
"has_audio_understanding",
"model_type",
"architectures",
"embedding",
}
```
docs_new/docs/basic_usage/openai_api_embeddings.mdx
+21 -3
View File
@@ -12,7 +12,7 @@ This tutorial covers the embedding APIs for embedding models. For a list of the
## Launch A Server
Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command.
Launch the server in your terminal and wait for it to initialize. Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models still require `--is-embedding`.
@@ -22,8 +22,8 @@ from sglang.utils import wait_for_server, print_highlight, terminate_process
embedding_process, port = launch_server_cmd(
"""
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
--host 0.0.0.0 --is-embedding --log-level warning
sglang serve --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
--is-embedding --log-level warning
"""
)
@@ -117,6 +117,24 @@ input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))[
print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
```
## Compact Base64 Responses
Set `encoding_format` to `base64` when JSON arrays would dominate response size. The encoded value contains little-endian FP32 values and can be decoded by OpenAI-compatible clients.
```python Example
response = requests.post(
f"http://localhost:{port}/v1/embeddings",
json={
"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct",
"input": text,
"encoding_format": "base64",
},
)
base64_embedding = response.json()["data"][0]["embedding"]
print_highlight(f"Base64 embedding: {base64_embedding[:20]}...")
```
```python Example
terminate_process(embedding_process)
docs_new/docs/developer_guide/bench_serving.mdx
+33
View File
@@ -16,6 +16,7 @@ This guide explains how to benchmark online serving throughput and latency using
- `sglang` / `sglang-native`: `POST /generate`
- `sglang-oai`, `vllm`, `lmdeploy`: `POST /v1/completions`
- `sglang-oai-chat`, `vllm-chat`, `lmdeploy-chat`: `POST /v1/chat/completions`
- `sglang-embedding`, `vllm-embedding`: `POST /v1/embeddings`
- `trt` (TensorRT-LLM): `POST /v2/models/ensemble/generate_stream`
- `gserver`: Custom server (Not Implemented yet in this script)
- `truss`: `POST /v1/models/model:predict`
@@ -55,6 +56,38 @@ python3 -m sglang.bench_serving \
--model meta-llama/Llama-3.1-8B-Instruct
```
### Fair embedding comparison
Use the two embedding backends with the same model, tokenizer, input length, prompt count, and concurrency. The benchmark reports input-token throughput and end-to-end latency; embeddings have no decode-side TTFT or TPOT.
```bash Command
# Start either server on the same hardware and precision, then run one at a time.
python3 -m sglang.bench_serving \
--backend sglang-embedding \
--model google/embeddinggemma-300m \
--dataset-name random \
--random-input-len 2048 \
--num-prompts 300 \
--max-concurrency 64 \
--warmup-requests 3 \
--flush-cache
```
```bash Command
# vLLM's cache reset endpoint requires VLLM_SERVER_DEV_MODE=1 at server startup.
python3 -m sglang.bench_serving \
--backend vllm-embedding \
--model google/embeddinggemma-300m \
--dataset-name random \
--random-input-len 2048 \
--num-prompts 300 \
--max-concurrency 64 \
--warmup-requests 3 \
--flush-cache
```
`--flush-cache` calls `/flush_cache` for SGLang and `/reset_prefix_cache` for vLLM after warmup. For vLLM, start the server with `VLLM_SERVER_DEV_MODE=1`; without it the benchmark fails loudly rather than accidentally measuring warm-cache performance.
### Datasets
Select with `--dataset-name`:
docs_new/docs/supported-models/embedding_models.mdx
+23 -13
View File
@@ -5,7 +5,7 @@ description: Dense and sparse embedding models with FlashInfer acceleration and
SGLang provides robust support for embedding models by integrating efficient serving mechanisms with its flexible programming interface. This integration allows for streamlined handling of embedding tasks, facilitating faster and more accurate retrieval and semantic search operations. SGLang's architecture enables better resource utilization and reduced latency in embedding model deployment.
<Warning>
Embedding models are executed with `--is-embedding` flag and some may require `--trust-remote-code`
Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models require `--is-embedding`; add `--trust-remote-code` when the model requires it.
</Warning>
## Quick Start
@@ -13,13 +13,21 @@ Embedding models are executed with `--is-embedding` flag and some may require `-
### Launch Server
```bash
python3 -m sglang.launch_server \
sglang serve \
--model-path Qwen/Qwen3-Embedding-4B \
--is-embedding \
--host 0.0.0.0 \
--port 30000
--is-embedding
```
### EmbeddingGemma
EmbeddingGemma uses bidirectional attention and is auto-detected, so its best default command is simply:
```bash
sglang serve --model-path google/embeddinggemma-300m
```
On CUDA, SGLang automatically uses breakable CUDA graph (BCG) for its full encoder prefill and disables incompatible radix-cache and chunked-prefill behavior. Do not add the deprecated piecewise CUDA graph knobs.
### Client Request
```python
@@ -30,7 +38,7 @@ url = "http://127.0.0.1:30000"
payload = {
"model": "Qwen/Qwen3-Embedding-4B",
"input": "What is the capital of France?",
"encoding_format": "float"
"encoding_format": "float" # or "base64" for compact FP32 responses
}
response = requests.post(url + "/v1/embeddings", json=payload).json()
@@ -43,12 +51,10 @@ print("Embedding:", response["data"][0]["embedding"])
For multimodal models like GME that support both text and images:
```bash
python3 -m sglang.launch_server \
sglang serve \
--model-path Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
--is-embedding \
--chat-template gme-qwen2-vl \
--host 0.0.0.0 \
--port 30000
--chat-template gme-qwen2-vl
```
```python Example
@@ -85,11 +91,9 @@ print("Embeddings:", [x.get("embedding") for x in response.get("data", [])])
If the model config already includes `matryoshka_dimensions` or `is_matryoshka` then no override is needed. Otherwise, you can use `--json-model-override-args` as below:
```bash Command
python3 -m sglang.launch_server \
sglang serve \
--model-path Qwen/Qwen3-Embedding-0.6B \
--is-embedding \
--host 0.0.0.0 \
--port 30000 \
--json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 1024, 1536]}'
```
@@ -133,6 +137,12 @@ print("Embedding:", response["data"][0]["embedding"])
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>EmbeddingGemma</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`google/embeddinggemma-300m`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Bidirectional Gemma3 text encoder; auto-detected and served with BCG by default</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>E5 (Llama/Mistral based)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`intfloat/e5-mistral-7b-instruct`</td>