minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[docs] major SGL Model Gateway documentation update (#15715)

Browse Source
This commit is contained in:
Simo Lin
2025-12-23 20:26:09 -08:00
committed by GitHub
parent ac320a6f04
commit 9665574937
4 changed files with 1882 additions and 490 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sgl-model-gateway/README.md
+227 -10
View File
@@ -31,15 +31,17 @@ High-performance model routing control and data plane for large-scale LLM deploy
- Multi-model HTTP serving and inference gateway routing with model-specific policies.
- Prefill/decode disaggregation, including bootstrap port handling and cache-aware merging.
- gRPC routing with fully Rust tokenizer loading, reasoning parser selection, and tool parser integration for OpenAI-compatible endpoints—supporting streaming and non-streaming modes across DeepSeek, Llama, Kimi K2, Qwen, GPT-OSS, Mistral, Step-3, GLM4, GLM4.7 and other reasoning-capable models.
- OpenAI-compatible `/v1/chat/completions`, `/v1/responses`, `/v1/conversations`, `/v1/embeddings`, and `/v1/rerank` endpoints.
- OpenAI-compatible `/v1/chat/completions`, `/v1/responses`, `/v1/conversations`, `/v1/embeddings`, `/v1/rerank`, `/v1/classify` endpoints.
- **Tokenization APIs**: HTTP endpoints for tokenize (`/v1/tokenize`) and detokenize (`/v1/detokenize`) with batch support; tokenizer management APIs for dynamic registration.
- **Parser endpoints**: Reasoning parser (`/parse/reasoning`) and function call parser (`/parse/function_call`) for separating reasoning content and extracting tool calls.
- Native MCP client integration supporting all MCP transport protocols (STDIO, HTTP, SSE, and Streamable) for tool execution loops.
- Pluggable history connectors: in-memory, disabled, or Oracle ATP (with pooling and credential support).
- Pluggable history connectors: in-memory, disabled, Oracle ATP, or PostgreSQL (with pooling and credential support).
- Reliability controls: retry with jitter, worker-scoped circuit breakers, token bucket limiter with optional queue, and cache flush APIs.
- Service discovery for regular and PD workloads with independent selectors.
- Prometheus metrics and structured tracing for every stage of routing.
- **Comprehensive observability**: 40+ Prometheus metrics across HTTP, router, worker, circuit breaker, retry, discovery, MCP, and database layers; OpenTelemetry tracing with OTLP export; structured logging with request ID propagation.
## Documentation
- **User Guide**: [docs.sglang.io/advanced_features/router.html](https://docs.sglang.io/advanced_features/router.html)
- **User Guide**: [docs.sglang.io/advanced_features/sgl_model_gateway.html](https://docs.sglang.io/advanced_features/sgl_model_gateway.html)
- Additional guides, API references, and deployment patterns are continuously updated alongside SGLang releases.
## Installation
@@ -476,11 +478,115 @@ The HTTP router exposes the full OpenAI-compatible surface area (`/generate`, `/
| `POST /v1/responses` | Create background responses, returns response IDs. |
| `GET /v1/responses/{id}` | Retrieve stored responses. |
| Conversation endpoints (`/v1/conversations`, `/v1/conversations/{id}`, `/v1/conversations/{id}/items`) | Manage chat history. |
| `POST /v1/embeddings` | Forward embedding requests. |
| `POST /v1/embeddings` | Forward embedding requests (HTTP and gRPC). |
| `POST /v1/rerank`, `POST /rerank` | Ranking APIs. |
| `POST /v1/classify` | Text classification endpoint. |
Public health endpoints (`/liveness`, `/readiness`, `/health`, `/health_generate`) reflect registry state; readiness ensures PD workers are paired and IGW has at least one healthy route.
### Tokenization Endpoints
The gateway provides HTTP endpoints for text tokenization, designed to mirror the SGLang Python tokenization API with support for batch operations.
| Endpoint | Method | Description |
|-------------------------------|----------|-------------------------------------------------------|
| `POST /v1/tokenize` | `POST` | Tokenize text to token IDs (single or batch). |
| `POST /v1/detokenize` | `POST` | Convert token IDs back to text (single or batch). |
| `POST /v1/tokenizers` | `POST` | Register a new tokenizer (async, returns job status). |
| `GET /v1/tokenizers` | `GET` | List all registered tokenizers. |
| `GET /v1/tokenizers/{id}` | `GET` | Get tokenizer info by UUID. |
| `GET /v1/tokenizers/{id}/status` | `GET` | Check async tokenizer loading status. |
| `DELETE /v1/tokenizers/{id}` | `DELETE` | Remove a tokenizer from the registry. |
**Tokenize Request:**
```json
{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Hello, world!"
}
```
**Batch Tokenize Request:**
```json
{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": ["Hello", "World", "How are you?"]
}
```
**Tokenize Response:**
```json
{
"tokens": [15339, 11, 1917, 0],
"count": 4,
"char_count": 13
}
```
**Detokenize Request:**
```json
{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"tokens": [15339, 11, 1917, 0],
"skip_special_tokens": true
}
```
**Add Tokenizer (async registration):**
```bash
# Register from HuggingFace
curl -X POST http://localhost:30000/v1/tokenizers \
-H "Content-Type: application/json" \
-d '{"name": "llama3", "source": "meta-llama/Llama-3.1-8B-Instruct"}'
# Check status
curl http://localhost:30000/v1/tokenizers/{tokenizer_id}/status
```
### Parser Endpoints
The gateway provides admin endpoints for parsing reasoning content and function calls from LLM outputs.
| Endpoint | Method | Description |
|--------------------------|--------|--------------------------------------------------------|
| `POST /parse/reasoning` | `POST` | Separate reasoning (`<think>`) from normal text. |
| `POST /parse/function_call` | `POST` | Parse function/tool calls from text. |
**Separate Reasoning Request:**
```json
{
"text": "<think>Let me analyze this step by step...</think>The answer is 42.",
"parser": "deepseek-r1"
}
```
**Response:**
```json
{
"normal_text": "The answer is 42.",
"reasoning_text": "Let me analyze this step by step..."
}
```
**Supported Reasoning Parsers:**
- `deepseek-r1` - DeepSeek-R1 (initial reasoning mode)
- `qwen3` - Qwen-3 models
- `qwen3-thinking` / `qwen-thinking` - Qwen thinking variant
- `kimi` - Kimi K2 with Unicode tokens
- `glm45` / `glm47` - GLM-4.5/4.6/4.7 models
- `step3` - Step-3 models
- `minimax` - MiniMax models
**Function Call Parsing:**
```json
{
"text": "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}",
"parser": "json"
}
```
Supported tool parsers: `json`, `python`, `xml`.
## Conversations, Responses, and Data Connectors
- `--history-backend memory` (default) stores responses and conversations in-process.
- `--history-backend none` disables persistence while keeping APIs.
@@ -566,11 +672,67 @@ Only one of `--oracle-dsn` or `--oracle-tns-alias` should be supplied.
Per-model overrides are available in PD mode (`--prefill-policy`, `--decode-policy`) and IGW mode via the worker registry.
## Observability
- **Logging**: Structured tracing through `tracing` with optional file sink (`--log-dir`) and `--log-level` (`debug`, `info`, `warn`, `error`).
- **Prometheus Metrics**: Enable with `--prometheus-host`/`--prometheus-port` (defaults to `0.0.0.0:29000`). Metrics cover request latency, retry behavior, circuit breaker states, worker health/load, queue depth, PD pipeline stats, tokenizer timings, and MCP activity.
- **Request IDs**: Configurable headers via `--request-id-headers`; responses include `x-request-id`.
- **CORS**: Set `--cors-allowed-origins` for browser access.
- **Request Tracing via OpenTelemetry**: Enable with `--enable-trace` and set opentelemetry collector endpoint with `--otlp-traces-endpoint <ip>:<port>`.
### Logging
Structured tracing through `tracing` with optional file sink (`--log-dir`) and `--log-level` (`debug`, `info`, `warn`, `error`).
### Prometheus Metrics
Enable with `--prometheus-host`/`--prometheus-port` (defaults to `0.0.0.0:29000`).
**Metric Categories (40+ metrics):**
| Layer | Metric Prefix | Description |
|-------|---------------|-------------|
| HTTP | `smg_http_*` | Request counts, duration, active connections, rate limiting |
| Router | `smg_router_*` | Requests by model/endpoint, latency, errors, upstream responses |
| Inference | `smg_router_ttft/tpot/tokens_*` | Time to first token, time per output token, token counts (gRPC) |
| Worker | `smg_worker_*` | Pool size, active connections, health checks, selection events |
| Circuit Breaker | `smg_worker_cb_*` | State (closed/open/half-open), transitions, outcomes |
| Retry | `smg_worker_retries_*` | Retry attempts, exhausted retries, backoff duration |
| Discovery | `smg_discovery_*` | K8s registrations, sync duration, workers discovered |
| MCP | `smg_mcp_*` | Tool calls, duration, active servers, iterations |
| Database | `smg_db_*` | Operations, duration, connections, items stored |
**Key Metrics:**
- `smg_router_ttft_seconds` - Time to first token histogram (gRPC mode)
- `smg_router_tpot_seconds` - Time per output token histogram (gRPC mode)
- `smg_router_tokens_total` - Total input/output tokens by model
- `smg_router_generation_duration_seconds` - End-to-end generation time
- `smg_worker_cb_state` - Circuit breaker state gauge (0=closed, 1=open, 2=half-open)
**Duration Buckets:**
1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 15s, 30s, 45s, 60s, 90s, 120s, 180s, 240s
### OpenTelemetry Tracing
Enable distributed tracing with OTLP export:
```bash
python -m sglang_router.launch_router \
--worker-urls http://worker1:8000 \
--enable-trace \
--otlp-traces-endpoint localhost:4317
```
**Features:**
- OTLP/gRPC exporter (default port 4317)
- W3C Trace Context propagation for HTTP and gRPC
- Batch span processing (500ms delay, 64 span batch size)
- Custom filtering to reduce noise (only exports relevant spans)
- Trace context injection into upstream worker requests
**Configuration:**
- `--enable-trace` - Enable OpenTelemetry tracing
- `--otlp-traces-endpoint <host:port>` - OTLP collector endpoint
### Request ID Propagation
Configure headers for request ID extraction:
```bash
--request-id-headers x-request-id x-trace-id x-correlation-id
```
Responses include `x-request-id` header for correlation.
### CORS
Set `--cors-allowed-origins` for browser access.
## Security
@@ -608,6 +770,61 @@ curl -X POST "http://localhost:8080/add_worker?url=http://worker3:8000&api_key=w
- Router logs a warning when a worker is registered without a key while the router expects authentication.
- When router and workers share the same key, still include the key when invoking dynamic registration APIs.
### TLS (HTTPS) for Gateway Server
Enable TLS to serve the gateway over HTTPS:
```bash
python3 -m sglang_router.launch_router \
--worker-urls http://worker1:8000 \
--tls-cert-path /path/to/server.crt \
--tls-key-path /path/to/server.key
```
| Parameter | Description |
|-----------|-------------|
| `--tls-cert-path` | Path to server certificate (PEM format) |
| `--tls-key-path` | Path to server private key (PEM format) |
Both parameters must be provided together. The gateway uses rustls with the ring crypto provider for TLS termination. If TLS is not configured, the gateway falls back to plain HTTP.
### mTLS for Worker Communication
Enable mutual TLS (mTLS) for secure communication with workers in HTTP mode:
```bash
python3 -m sglang_router.launch_router \
--worker-urls https://worker1:8443 https://worker2:8443 \
--client-cert-path /path/to/client.crt \
--client-key-path /path/to/client.key \
--ca-cert-path /path/to/ca.crt
```
| Parameter | Description |
|-----------|-------------|
| `--client-cert-path` | Path to client certificate for mTLS (PEM format) |
| `--client-key-path` | Path to client private key for mTLS (PEM format) |
| `--ca-cert-path` | Path to CA certificate for verifying worker TLS (PEM format) |
**Key Points:**
- Client certificate and key must be provided together
- Multiple CA certificates can be added with multiple `--ca-cert-path` flags
- Uses rustls backend when TLS is configured
- Single HTTP client is created for all workers (assumes single security domain)
- TCP keepalive (30 seconds) is enabled for long-lived connections
**Full TLS Example (Gateway HTTPS + Worker mTLS):**
```bash
python3 -m sglang_router.launch_router \
--worker-urls https://worker1:8443 https://worker2:8443 \
--tls-cert-path /etc/certs/server.crt \
--tls-key-path /etc/certs/server.key \
--client-cert-path /etc/certs/client.crt \
--client-key-path /etc/certs/client.key \
--ca-cert-path /etc/certs/ca.crt \
--api-key "secure-api-key"
```
## Development & Testing
```bash
# Build Rust components (debug mode, fast)