[Router] Add bucket-aware policy domains and native cache indexing (#38108)
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> Co-authored-by: inkcherry <mingzhi.liu@amd.com> Co-authored-by: yangbodong22011 <13137470+yangbodong22011@users.noreply.github.com>
This commit is contained in:
co-authored by
inkcherry
yangbodong22011
parent
a176ba2f7b
commit
5bebe7a033
@@ -133,7 +133,8 @@ sgl-router \
|
||||
--model-id <model-id> \
|
||||
--tokenizer-path <huggingface-repo-or-tokenizer> \
|
||||
--worker-urls http://127.0.0.1:30000 \
|
||||
--policy cache_aware_zmq \
|
||||
--policy cache_aware \
|
||||
--cache-prefix-provider indexer \
|
||||
--kv-indexer-endpoint http://127.0.0.1:50051 \
|
||||
--kv-indexer-query-timeout-ms 100 \
|
||||
--kv-indexer-query-max-inflight 32
|
||||
|
||||
Reference in New Issue
Block a user