[Feature] Add process-local in-memory KV indexer and Router integration (#33370)
Co-authored-by: Wu, Yutong <yutong.wu@amd.com> Co-authored-by: TianDi101 <ditian12@amd.com> Co-authored-by: Zhangheng <hzh0425@apache.org>
This commit is contained in:
co-authored by
Wu, Yutong
TianDi101
Zhangheng
parent
238ba40c27
commit
360d10d6bc
@@ -50,6 +50,26 @@ Omit `--service-discovery-namespace` to watch all namespaces (requires
|
||||
cluster-wide RBAC). For prefill/decode disaggregation, replace `--selector`
|
||||
with `--prefill-selector` and `--decode-selector`.
|
||||
|
||||
External KV indexer as the cache-aware signal source:
|
||||
|
||||
```bash
|
||||
sgl-router \
|
||||
--model-id qwen3 \
|
||||
--tokenizer-path /models/qwen3/tokenizer.json \
|
||||
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
|
||||
--policy cache_aware_zmq \
|
||||
--kv-indexer-endpoint http://10.0.0.10:50051 \
|
||||
--kv-indexer-query-timeout-ms 100 \
|
||||
--kv-indexer-query-max-inflight 32
|
||||
```
|
||||
|
||||
The existing cache-aware policy and thresholds are reused. When configured, the
|
||||
Indexer replaces the Router-local radix tree as the cache signal. A successful
|
||||
query with no usable match selects by minimum active load; connection failures,
|
||||
timeouts, local admission rejection, and server rejection fail the Router
|
||||
request with `503` rather than silently switching signals. The timeout and local
|
||||
concurrency bound default to 100ms and 32 respectively.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0.
|
||||
|
||||
Reference in New Issue
Block a user