5bebe7a033
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> Co-authored-by: inkcherry <mingzhi.liu@amd.com> Co-authored-by: yangbodong22011 <13137470+yangbodong22011@users.noreply.github.com>
sgl-router
Slim, KV-aware, OpenAI-compatible router for SGLang workers.
Serves a single model and routes across its workers. Exposes
/v1/tokenize, /v1/detokenize, /v1/models, /v1/chat/completions
(buffered and SSE), plus /healthz / /readyz and /metrics. Worker
pools come from either a static URL list or Kubernetes EndpointSlice
discovery.
Building
cd experimental/sgl-router
cargo build --release
Running
The router is configured entirely through CLI flags (run
sgl-router --help for the full list). It serves exactly one model, so
--model-id is required, along with exactly one discovery backend.
--tokenizer-path is optional: give it a local tokenizer.json path or a
HuggingFace repo id, and when omitted the router downloads the tokenizer
for --model-id from HuggingFace (honoring HF_TOKEN / HF_HOME).
Static worker list:
sgl-router \
--host 0.0.0.0 --port 30000 \
--model-id qwen3 \
--tokenizer-path /models/qwen3/tokenizer.json \
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000
Kubernetes EndpointSlice discovery:
sgl-router \
--host 0.0.0.0 --port 30000 \
--model-id qwen3 \
--tokenizer-path /models/qwen3/tokenizer.json \
--service-discovery \
--service-discovery-namespace prod \
--selector app=engines-qwen3
Omit --service-discovery-namespace to watch all namespaces (requires
cluster-wide RBAC). For prefill/decode disaggregation, replace --selector
with --prefill-selector and --decode-selector.
External KV indexer as the cache-aware signal source:
sgl-router \
--model-id qwen3 \
--tokenizer-path /models/qwen3/tokenizer.json \
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
--policy cache_aware \
--cache-prefix-provider indexer \
--kv-indexer-endpoint http://10.0.0.10:50051 \
--kv-indexer-query-timeout-ms 100 \
--kv-indexer-query-max-inflight 32
The Indexer replaces the Router-local radix tree as the native Cache-Aware signal. Query timeouts and local concurrency are bounded by the two Indexer options, which default to 100 ms and 32 respectively.
Upgrading from cache_aware_zmq
The cache_aware_zmq policy has been removed. Configurations using it should
select --policy cache_aware and choose a native cache-prefix source: the
Router-local radix tree (the default), or the external Indexer shown above.
The legacy --cache-threshold, --balance-abs-threshold, and
--balance-rel-threshold flags have also been removed. They do not have
one-to-one replacements; remove them and review the current sgl-router --help output when tuning Cache-Aware routing.
License
Apache-2.0.