[Docs] Sync docs_new with legacy docs and update migration redirects (#23337)
Co-authored-by: Mingyi <wisclmy0611@gmail.com>
This commit is contained in:
@@ -7,7 +7,7 @@ Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its nati
|
||||
|
||||
- `/generate` (text generation model)
|
||||
- `/get_model_info`
|
||||
- `/get_server_info`
|
||||
- `/server_info`
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/flush_cache`
|
||||
@@ -35,7 +35,7 @@ server_process, port = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
wait_for_server(f"http://localhost:{port}", process=server_process)
|
||||
```
|
||||
|
||||
## Generate (text generation model)
|
||||
@@ -96,7 +96,7 @@ Gets the server information including CLI arguments, token limits, and memory po
|
||||
- `get_max_total_num_tokens`
|
||||
|
||||
```python Example
|
||||
url = f"http://localhost:{port}/get_server_info"
|
||||
url = f"http://localhost:{port}/server_info"
|
||||
|
||||
response = requests.get(url)
|
||||
print_highlight(response.text)
|
||||
@@ -124,6 +124,14 @@ print_highlight(response.text)
|
||||
|
||||
Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API.
|
||||
|
||||
Parameters:
|
||||
- `timeout` (query, float, default `0`, unit: seconds): Wait time for idle state before flushing. `0` means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors.
|
||||
|
||||
```bash Command
|
||||
# With timeout (wait up to 30s for idle state)
|
||||
curl -s -X POST "http://127.0.0.1:30000/flush_cache?timeout=30"
|
||||
```
|
||||
|
||||
```python Example
|
||||
url = f"http://localhost:{port}/flush_cache"
|
||||
|
||||
@@ -176,14 +184,12 @@ Encode text into embeddings. Note that this API is only available for [embedding
|
||||
Therefore, we launch a new server to server an embedding model.
|
||||
|
||||
```python Example
|
||||
embedding_process, port = launch_server_cmd(
|
||||
"""
|
||||
embedding_process, port = launch_server_cmd("""
|
||||
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
|
||||
--host 0.0.0.0 --is-embedding --log-level warning
|
||||
"""
|
||||
)
|
||||
""")
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
wait_for_server(f"http://localhost:{port}", process=embedding_process)
|
||||
```
|
||||
|
||||
```python Example
|
||||
@@ -205,14 +211,12 @@ terminate_process(embedding_process)
|
||||
Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.
|
||||
|
||||
```python Example
|
||||
reranker_process, port = launch_server_cmd(
|
||||
"""
|
||||
reranker_process, port = launch_server_cmd("""
|
||||
python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \
|
||||
--host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning
|
||||
"""
|
||||
)
|
||||
""")
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
wait_for_server(f"http://localhost:{port}", process=reranker_process)
|
||||
```
|
||||
|
||||
```python Example
|
||||
@@ -253,14 +257,12 @@ Parameters:
|
||||
The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`.
|
||||
|
||||
```python Example
|
||||
score_process, port = launch_server_cmd(
|
||||
"""
|
||||
score_process, port = launch_server_cmd("""
|
||||
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
|
||||
--host 0.0.0.0 --log-level warning
|
||||
"""
|
||||
)
|
||||
""")
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
wait_for_server(f"http://localhost:{port}", process=score_process)
|
||||
```
|
||||
|
||||
```python Example
|
||||
@@ -297,13 +299,11 @@ SGLang Runtime also supports reward models. Here we use a reward model to classi
|
||||
# Note that SGLang now treats embedding models and reward models as the same type of models.
|
||||
# This will be updated in the future.
|
||||
|
||||
reward_process, port = launch_server_cmd(
|
||||
"""
|
||||
reward_process, port = launch_server_cmd("""
|
||||
python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning
|
||||
"""
|
||||
)
|
||||
""")
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
wait_for_server(f"http://localhost:{port}", process=reward_process)
|
||||
```
|
||||
|
||||
```python Example
|
||||
@@ -347,7 +347,7 @@ expert_record_server_process, port = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
wait_for_server(f"http://localhost:{port}", process=expert_record_server_process)
|
||||
```
|
||||
|
||||
```python Example
|
||||
@@ -376,13 +376,11 @@ terminate_process(expert_record_server_process)
|
||||
This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization.
|
||||
|
||||
```python Example
|
||||
tokenizer_free_server_process, port = launch_server_cmd(
|
||||
"""
|
||||
tokenizer_free_server_process, port = launch_server_cmd("""
|
||||
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct
|
||||
"""
|
||||
)
|
||||
""")
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
wait_for_server(f"http://localhost:{port}", process=tokenizer_free_server_process)
|
||||
```
|
||||
|
||||
```python Example
|
||||
|
||||
Reference in New Issue
Block a user