minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[docs] Fix non-clickable ToC links in model gateway documentation (#16054)

Browse Source
This commit is contained in:
Simo Lin
2025-12-28 23:35:37 -08:00
committed by GitHub
parent f784cbfa92
commit de2799f3f5
1 changed files with 20 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/sgl_model_gateway.md
+20 -20
View File
@@ -10,38 +10,38 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale
2. [Architecture](#architecture) 2. [Architecture](#architecture)
- [Control Plane](#control-plane) - [Control Plane](#control-plane)
- [Data Plane](#data-plane) - [Data Plane](#data-plane)
- [Storage & Privacy](#storage--privacy) - [Storage and Privacy](#storage-and-privacy)
3. [Installation](#installation) 3. [Installation](#installation)
4. [Quick Start](#quick-start) 4. [Quick Start](#quick-start)
5. [Deployment Modes](#deployment-modes) 5. [Deployment Modes](#deployment-modes)
- [Co-launch Router + Workers](#co-launch-router--workers) - [Co-launch Router and Workers](#co-launch-router-and-workers)
- [Separate Launch (HTTP)](#separate-launch-http) - [Separate Launch (HTTP)](#separate-launch-http)
- [gRPC Launch](#grpc-launch) - [gRPC Launch](#grpc-launch)
- [Prefill/Decode Disaggregation](#prefilldecode-disaggregation) - [Prefill-Decode Disaggregation](#prefill-decode-disaggregation)
- [OpenAI Backend Proxy](#openai-backend-proxy) - [OpenAI Backend Proxy](#openai-backend-proxy)
- [Multi-Model Inference Gateway](#multi-model-inference-gateway) - [Multi-Model Inference Gateway](#multi-model-inference-gateway)
6. [API Reference](#api-reference) 6. [API Reference](#api-reference)
- [Inference Endpoints](#inference-endpoints) - [Inference Endpoints](#inference-endpoints)
- [Tokenization Endpoints](#tokenization-endpoints) - [Tokenization Endpoints](#tokenization-endpoints)
- [Parser Endpoints](#parser-endpoints) - [Parser Endpoints](#parser-endpoints)
- [Conversation & Response APIs](#conversation--response-apis) - [Conversation and Response APIs](#conversation-and-response-apis)
- [Worker Management APIs](#worker-management-apis) - [Worker Management APIs](#worker-management-apis)
- [Admin & Health Endpoints](#admin--health-endpoints) - [Admin and Health Endpoints](#admin-and-health-endpoints)
7. [Load Balancing Policies](#load-balancing-policies) 7. [Load Balancing Policies](#load-balancing-policies)
8. [Reliability & Flow Control](#reliability--flow-control) 8. [Reliability and Flow Control](#reliability-and-flow-control)
- [Retries](#retries) - [Retries](#retries)
- [Circuit Breaker](#circuit-breaker) - [Circuit Breaker](#circuit-breaker)
- [Rate Limiting & Queuing](#rate-limiting--queuing) - [Rate Limiting and Queuing](#rate-limiting-and-queuing)
- [Health Checks](#health-checks) - [Health Checks](#health-checks)
9. [Reasoning Parser Integration](#reasoning-parser-integration) 9. [Reasoning Parser Integration](#reasoning-parser-integration)
10. [Tool Call Parsing](#tool-call-parsing) 10. [Tool Call Parsing](#tool-call-parsing)
11. [Tokenizer Management](#tokenizer-management) 11. [Tokenizer Management](#tokenizer-management)
12. [MCP Integration](#mcp-integration) 12. [MCP Integration](#mcp-integration)
13. [Service Discovery (Kubernetes)](#service-discovery-kubernetes) 13. [Service Discovery (Kubernetes)](#service-discovery-kubernetes)
14. [History & Data Connectors](#history--data-connectors) 14. [History and Data Connectors](#history-and-data-connectors)
15. [WASM Middleware](#wasm-middleware) 15. [WASM Middleware](#wasm-middleware)
16. [Language Bindings](#language-bindings) 16. [Language Bindings](#language-bindings)
17. [Security & Authentication](#security--authentication) 17. [Security and Authentication](#security-and-authentication)
- [TLS (HTTPS) for Gateway Server](#tls-https-for-gateway-server) - [TLS (HTTPS) for Gateway Server](#tls-https-for-gateway-server)
- [mTLS for Worker Communication](#mtls-for-worker-communication) - [mTLS for Worker Communication](#mtls-for-worker-communication)
18. [Observability](#observability) 18. [Observability](#observability)
@@ -49,7 +49,7 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale
- [OpenTelemetry Tracing](#opentelemetry-tracing) - [OpenTelemetry Tracing](#opentelemetry-tracing)
- [Logging](#logging) - [Logging](#logging)
19. [Production Recommendations](#production-recommendations) 19. [Production Recommendations](#production-recommendations)
- [Security](#security-1) - [Security Best Practices](#security-best-practices)
- [High Availability](#high-availability) - [High Availability](#high-availability)
- [Performance](#performance) - [Performance](#performance)
- [Kubernetes Deployment](#kubernetes-deployment) - [Kubernetes Deployment](#kubernetes-deployment)
@@ -88,7 +88,7 @@ SGLang Model Gateway is a high-performance model-routing gateway for large-scale
- **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings. - **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings.
- **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local. - **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local.
### Storage & Privacy ### Storage and Privacy
- Conversation and response history is stored at the router tier (memory, none, Oracle ATP, or PostgreSQL). The same history can power multiple models or MCP loops without sending data to upstream vendors. - Conversation and response history is stored at the router tier (memory, none, Oracle ATP, or PostgreSQL). The same history can power multiple models or MCP loops without sending data to upstream vendors.
- `/v1/responses` agentic flows, MCP sessions, and conversation APIs share the same storage layer, enabling compliance for regulated workloads. - `/v1/responses` agentic flows, MCP sessions, and conversation APIs share the same storage layer, enabling compliance for regulated workloads.
@@ -170,7 +170,7 @@ python -m sglang_router.launch_router \
## Deployment Modes ## Deployment Modes
### Co-launch Router + Workers ### Co-launch Router and Workers
Launch the router and a fleet of SGLang workers in one process: Launch the router and a fleet of SGLang workers in one process:
@@ -238,7 +238,7 @@ python -m sglang_router.launch_router \
The gRPC router supports both regular HTTP-equivalent serving and PD (prefill/decode) serving. Provide `--tokenizer-path` or `--model-path` (HuggingFace ID or local directory) whenever connection mode resolves to gRPC. The gRPC router supports both regular HTTP-equivalent serving and PD (prefill/decode) serving. Provide `--tokenizer-path` or `--model-path` (HuggingFace ID or local directory) whenever connection mode resolves to gRPC.
### Prefill/Decode Disaggregation ### Prefill-Decode Disaggregation
Split prefill and decode workers for PD-aware caching and balancing: Split prefill and decode workers for PD-aware caching and balancing:
@@ -420,7 +420,7 @@ The gateway provides admin endpoints for parsing reasoning content and function
} }
``` ```
### Conversation & Response APIs ### Conversation and Response APIs
| Method | Path | Description | | Method | Path | Description |
|--------|------|-------------| |--------|------|-------------|
@@ -487,7 +487,7 @@ Response:
} }
``` ```
### Admin & Health Endpoints ### Admin and Health Endpoints
| Method | Path | Description | | Method | Path | Description |
|--------|------|-------------| |--------|------|-------------|
@@ -537,7 +537,7 @@ Response:
--- ---
## Reliability & Flow Control ## Reliability and Flow Control
### Retries ### Retries
@@ -590,7 +590,7 @@ python -m sglang_router.launch_router \
- **Open**: Failing, requests rejected immediately - **Open**: Failing, requests rejected immediately
- **Half-Open**: Testing recovery, limited requests allowed - **Half-Open**: Testing recovery, limited requests allowed
### Rate Limiting & Queuing ### Rate Limiting and Queuing
```bash ```bash
python -m sglang_router.launch_router \ python -m sglang_router.launch_router \
@@ -796,7 +796,7 @@ Prefill pods can expose bootstrap ports via the `sglang.ai/bootstrap-port` annot
--- ---
## History & Data Connectors ## History and Data Connectors
| Backend | Description | Usage | | Backend | Description | Usage |
|---------|-------------|-------| |---------|-------------|-------|
@@ -1042,7 +1042,7 @@ go test -tags=integration -v ./...
--- ---
## Security & Authentication ## Security and Authentication
### Router API Key ### Router API Key
@@ -1210,7 +1210,7 @@ Responses include `x-request-id` header for correlation.
This section provides guidance for deploying SGLang Model Gateway in production environments. This section provides guidance for deploying SGLang Model Gateway in production environments.
### Security ### Security Best Practices
**Always enable TLS in production:** **Always enable TLS in production:**