sglang/experimental/sgl-router

sgl-router

Slim, KV-aware, OpenAI-compatible router for SGLang workers.

Serves a single model and routes across its workers. Exposes /v1/tokenize, /v1/detokenize, /v1/models, /v1/chat/completions (buffered and SSE), plus /healthz / /readyz and /metrics. Worker pools come from either a static URL list or Kubernetes EndpointSlice discovery.

Building

cd experimental/sgl-router
cargo build --release

Running

The router is configured entirely through CLI flags (run sgl-router --help for the full list). It serves exactly one model, so --model-id is required, along with exactly one discovery backend. --tokenizer-path is optional: give it a local tokenizer.json path or a HuggingFace repo id, and when omitted the router downloads the tokenizer for --model-id from HuggingFace (honoring HF_TOKEN / HF_HOME).

Static worker list:

sgl-router \
  --host 0.0.0.0 --port 30000 \
  --model-id qwen3 \
  --tokenizer-path /models/qwen3/tokenizer.json \
  --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000

Kubernetes EndpointSlice discovery:

sgl-router \
  --host 0.0.0.0 --port 30000 \
  --model-id qwen3 \
  --tokenizer-path /models/qwen3/tokenizer.json \
  --service-discovery \
  --service-discovery-namespace prod \
  --selector app=engines-qwen3

Omit --service-discovery-namespace to watch all namespaces (requires cluster-wide RBAC). For prefill/decode disaggregation, replace --selector with --prefill-selector and --decode-selector.

External KV indexer as the cache-aware signal source:

sgl-router \
  --model-id qwen3 \
  --tokenizer-path /models/qwen3/tokenizer.json \
  --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
  --policy cache_aware \
  --cache-prefix-provider indexer \
  --kv-indexer-endpoint http://10.0.0.10:50051 \
  --kv-indexer-query-timeout-ms 100 \
  --kv-indexer-query-max-inflight 32

The Indexer replaces the Router-local radix tree as the native Cache-Aware signal. Query timeouts and local concurrency are bounded by the two Indexer options, which default to 100 ms and 32 respectively.

Upgrading from cache_aware_zmq

The cache_aware_zmq policy has been removed. Configurations using it should select --policy cache_aware and choose a native cache-prefix source: the Router-local radix tree (the default), or the external Indexer shown above.

The legacy --cache-threshold, --balance-abs-threshold, and --balance-rel-threshold flags have also been removed. They do not have one-to-one replacements; remove them and review the current sgl-router --help output when tuning Cache-Aware routing.

License

Apache-2.0.