minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[router] Configure experimental sgl-router via CLI flags instead of a config file (#27073)

Browse Source 
Signed-off-by: Kangyan Zhou <zky314343421@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Kangyan-Zhou
2026-06-05 10:02:10 +08:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 631db6c757
commit bcf89928b4
43 changed files with 2355 additions and 1237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
experimental/sgl-router/README.md
+40 -5
View File
@@ -2,11 +2,11 @@
Slim, KV-aware, OpenAI-compatible router for SGLang workers.
**Status:** functional single-worker HTTP proxy. Exposes `/v1/tokenize`,
`/v1/detokenize`, `/v1/models`, `/v1/chat/completions` (buffered and SSE),
plus `/healthz` / `/readyz`. Forwards to one configured worker via reqwest;
parity-tested against `transformers.AutoTokenizer`. Multi-worker routing,
service discovery, and observability still pending.
Serves a single model and routes across its workers. Exposes
`/v1/tokenize`, `/v1/detokenize`, `/v1/models`, `/v1/chat/completions`
(buffered and SSE), plus `/healthz` / `/readyz` and `/metrics`. Worker
pools come from either a static URL list or Kubernetes EndpointSlice
discovery.
## Building
@@ -15,6 +15,41 @@ cd experimental/sgl-router
cargo build --release
```
## Running
The router is configured entirely through CLI flags (run
`sgl-router --help` for the full list). It serves exactly one model, so
`--model-id` is required, along with exactly one discovery backend.
`--tokenizer-path` is optional: give it a local `tokenizer.json` path or a
HuggingFace repo id, and when omitted the router downloads the tokenizer
for `--model-id` from HuggingFace (honoring `HF_TOKEN` / `HF_HOME`).
Static worker list:
```bash
sgl-router \
--host 0.0.0.0 --port 30000 \
--model-id qwen3 \
--tokenizer-path /models/qwen3/tokenizer.json \
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000
```
Kubernetes EndpointSlice discovery:
```bash
sgl-router \
--host 0.0.0.0 --port 30000 \
--model-id qwen3 \
--tokenizer-path /models/qwen3/tokenizer.json \
--service-discovery \
--service-discovery-namespace prod \
--selector app=engines-qwen3
```
Omit `--service-discovery-namespace` to watch all namespaces (requires
cluster-wide RBAC). For prefill/decode disaggregation, replace `--selector`
with `--prefill-selector` and `--decode-selector`.
## License
Apache-2.0.