embedding: centralize capabilities and complete OpenAI compatibility (#32481)

This commit is contained in:
Mick
2026-07-30 10:28:52 +08:00
committed by GitHub
parent 313a518bee
commit 22faf9fef8
16 changed files with 728 additions and 28 deletions
+2
View File
@@ -64,6 +64,7 @@ Get the information of the model.
- `has_audio_understanding`: Whether the model has audio-understanding capability.
- `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama").
- `architectures`: The model architectures from the HuggingFace config (e.g., ["Qwen2ForCausalLM"]).
- `embedding`: The resolved embedding-serving plan. It includes pooling, normalization, execution and attention style, Matryoshka dimensions, cache policy, and effective BCG prefill settings. This field is available when the model configuration exposes an embedding capability contract.
```python Example
url = f"http://localhost:{port}/get_model_info"
@@ -85,6 +86,7 @@ assert response_json.keys() == {
"has_audio_understanding",
"model_type",
"architectures",
"embedding",
}
```
@@ -12,7 +12,7 @@ This tutorial covers the embedding APIs for embedding models. For a list of the
## Launch A Server
Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command.
Launch the server in your terminal and wait for it to initialize. Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models still require `--is-embedding`.
@@ -22,8 +22,8 @@ from sglang.utils import wait_for_server, print_highlight, terminate_process
embedding_process, port = launch_server_cmd(
"""
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
--host 0.0.0.0 --is-embedding --log-level warning
sglang serve --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
--is-embedding --log-level warning
"""
)
@@ -117,6 +117,24 @@ input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))[
print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
```
## Compact Base64 Responses
Set `encoding_format` to `base64` when JSON arrays would dominate response size. The encoded value contains little-endian FP32 values and can be decoded by OpenAI-compatible clients.
```python Example
response = requests.post(
f"http://localhost:{port}/v1/embeddings",
json={
"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct",
"input": text,
"encoding_format": "base64",
},
)
base64_embedding = response.json()["data"][0]["embedding"]
print_highlight(f"Base64 embedding: {base64_embedding[:20]}...")
```
```python Example
terminate_process(embedding_process)