embedding: centralize capabilities and complete OpenAI compatibility (#32481)
This commit is contained in:
@@ -64,6 +64,7 @@ Get the information of the model.
|
||||
- `has_audio_understanding`: Whether the model has audio-understanding capability.
|
||||
- `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama").
|
||||
- `architectures`: The model architectures from the HuggingFace config (e.g., ["Qwen2ForCausalLM"]).
|
||||
- `embedding`: The resolved embedding-serving plan. It includes pooling, normalization, execution and attention style, Matryoshka dimensions, cache policy, and effective BCG prefill settings. This field is available when the model configuration exposes an embedding capability contract.
|
||||
|
||||
```python Example
|
||||
url = f"http://localhost:{port}/get_model_info"
|
||||
@@ -85,6 +86,7 @@ assert response_json.keys() == {
|
||||
"has_audio_understanding",
|
||||
"model_type",
|
||||
"architectures",
|
||||
"embedding",
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ This tutorial covers the embedding APIs for embedding models. For a list of the
|
||||
|
||||
## Launch A Server
|
||||
|
||||
Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command.
|
||||
Launch the server in your terminal and wait for it to initialize. Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models still require `--is-embedding`.
|
||||
|
||||
|
||||
|
||||
@@ -22,8 +22,8 @@ from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
embedding_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
|
||||
--host 0.0.0.0 --is-embedding --log-level warning
|
||||
sglang serve --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
|
||||
--is-embedding --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
@@ -117,6 +117,24 @@ input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))[
|
||||
print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
|
||||
```
|
||||
|
||||
## Compact Base64 Responses
|
||||
|
||||
Set `encoding_format` to `base64` when JSON arrays would dominate response size. The encoded value contains little-endian FP32 values and can be decoded by OpenAI-compatible clients.
|
||||
|
||||
```python Example
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/v1/embeddings",
|
||||
json={
|
||||
"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct",
|
||||
"input": text,
|
||||
"encoding_format": "base64",
|
||||
},
|
||||
)
|
||||
|
||||
base64_embedding = response.json()["data"][0]["embedding"]
|
||||
print_highlight(f"Base64 embedding: {base64_embedding[:20]}...")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(embedding_process)
|
||||
|
||||
Reference in New Issue
Block a user