[sgl-router] refactor - cache-aware policy (#40366)

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

experimental/sgl-router/POLICY_DESIGN.md

@@ -172,7 +172,7 @@ bucket, both groups share this one request-length decision. Policy fallback on
 a cache/affinity miss stays within that group's candidates. There is no second
 pass with relaxed admission and no post-policy substitution.
 
-SLO ordering, cache lookup, global session modes, and sticky policies are follow-ups. Their
+SLO ordering, global session modes, and sticky policies are follow-ups. Their
 integration must preserve bucket-first selection and the same-bucket PD rule.
 Cross-bucket affinity probing is not part of this interface. Session/routing
 keys still pass through `PickRequest` for policies operating inside the selected
@@ -568,7 +568,17 @@ Implemented here:
   dispatches only after one complete selection. Exhaustion retains admission reasons.
 - `AppContext::chat_routing` configures legacy versus reorg routing on the same
   endpoint and carries the reorg model-resolver map.
-
+- `CacheAwarePolicy` reads local radix-tree or remote indexer prefixes, intersects
+  exact worker URLs with the current group, applies hit thresholds and candidate
+  bounds, and preserves the soft queue gate, saturation pin and pressure guard.
+- `PrefixMemo` shares lookup results (including misses and unavailable backends)
+  across bucket attempts for one prepared request. Entries are keyed by the shared
+  `Arc<CacheSource>` so different index namespaces remain independent. Each pick
+  reruns its own candidate filtering and admission after obtaining a fresh snapshot.
+- Cache selection checks bounded candidates explicitly; hard rejection cannot
+  become a cold fallback or bypass admission through saturation pinning. A miss
+  defaults to power-of-two within the group's soft queue tier, then the cache
+  policy checks its fallback winner. Cache policies require plain/prefill groups.
 - `SessionAwarePolicy` reuses admitted model/bucket/role-scoped bindings from a
   shared `AffinityStore`, falling back to power-of-two for new or keyless sessions.
   Assignments follow admission; concurrent binding winners are rechecked.
@@ -576,8 +586,8 @@ Implemented here:
   The caller owns expiry and sweeper lifecycle. A binding may remain after a
   later PD group fails, because it records placement rather than dispatch.
 
-Follow-up order: cache-aware selection (#40366), concrete admission (#40271),
-then bucket SLO ordering, remaining policies, and production configuration.
+Follow-up order: concrete admission (#40271), then bucket SLO ordering
+in a separate PR, followed by remaining policies and production configuration.
 
 Not yet implemented in the reorg path:
 
@@ -586,7 +596,7 @@ Not yet implemented in the reorg path:
 - CLI/configuration parsing, validation, and model-specific construction.
   The YAML above is illustrative; reorg resolvers are installed in code.
 - Global session modes and sticky routing-key affinity.
-- Prefix memoization and cache-aware selection.
+- Power-of-k cache-miss fallback configuration and cache decision metrics.
 - Shared load interpretation, dispatch correction, and policy-specific
   dispatch-timestamp requirements.
 - PD compatibility filtering, retry integration, and legacy-route switchover.