diff --git a/experimental/sgl-router/ARCHITECTURE.md b/experimental/sgl-router/ARCHITECTURE.md
new file mode 100644
index 0000000000..b50a20af91
--- /dev/null
+++ b/experimental/sgl-router/ARCHITECTURE.md
@@ -0,0 +1,236 @@
+# SGLang router architecture
+
+This document describes `--chat-routing reorg`: the deployment, request flow,
+and routing policies. The default legacy path is outside its scope.
+
+- [1. Overview](#1-overview)
+- [2. Configuration examples](#2-configuration-examples)
+
+## 1. Overview
+
+### 1.1 Deployment Architecture
+
+A deployment consists of a **router** and **buckets of SGLang engines**. A bucket
+is a logical group defined in the router configuration. It contains either a
+**unified engine group**, where each engine handles both prefill and decode,
+or separate **prefill and decode (P/D) groups**.
+
+The example below shows short-context and long-context buckets, each with prefill
+and decode engines.
+
+
+
+Each engine publishes load reports and KV events, which the router uses for
+routing. An optional [external KV indexer](sgl-kv-indexer/README.md) can supply
+prefix matches. The router also maintains session bindings and in-flight request
+counts.
+
+To serve a request, the router chooses a bucket, selects engines within it, and
+forwards the request. In P/D, it sends the request to both the prefill and decode
+engines; the prefill engine transfers KV directly to the decode engine.
+
+### 1.2 Request routing lifecycle
+Routing a request typically goes through **preprocessing** (rendering and
+tokenizing), **picking a bucket**, **scoring candidate engines** (load, prefix
+matching, or session affinity), **choosing engine(s)**, **forwarding** the
+request, and **handling output tokens** (e.g. tool parsing). Along the way it
+also applies **admission control**, **failure handling**, and **retries**.
+
+### 1.3 Example: load-aware routing
+
+The router tries each **bucket** that fits the request's length, ordered by SLO
+preference, capacity, then `rank`; a request that fits no bucket is rejected
+with **400** before any engine is considered. In each bucket it drops the
+engines that fail **admission**, then picks the **less loaded of two engines
+sampled** from those that remain; when no engine admits the request it moves to
+the next bucket, and returns **503** when no bucket has one. If **dispatch**
+fails before any response and **retries** remain, it excludes that engine and
+restarts from the first bucket.
+
+**Load** is a vector of engine metrics compared field by field, and the first
+difference decides. For **unified and prefill** engines the order is estimated
+prefill queue time, waiting uncached tokens, waiting requests, running requests,
+then in-flight requests; **decode** engines rank by waiting and running requests,
+then KV usage.
+
+```mermaid
+%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#eaf3fb", "primaryTextColor": "#172b4d", "primaryBorderColor": "#345779", "lineColor": "#345779", "edgeLabelBackground": "#ffffff"}, "flowchart": {"curve": "stepAfter", "rankSpacing": 30, "nodeSpacing": 30}}}%%
+flowchart TD
+ R([Request]) --> L{"0. Fits any bucket's
length limits?"}
+ L -->|no| Y(["400 to client"])
+ L -->|yes| B["1. Take next compatible bucket"]
+ B --> A{"2. Any engine can admit
the request?"}
+ A -->|yes| S["3. Sample two admitted engines
Pick the less loaded one"]
+ S --> D["4. Dispatch to engine"]
+ A -->|no| M{"More buckets?"}
+ M -->|no| X(["503 to client"])
+ D -->|success, or
out of retries| E([Response to client])
+ D -->|failed before response,
retries left| T["Exclude failed engine"]
+ M -.->|yes: next bucket| B
+ T -.->|restart at first bucket| B
+ classDef next fill:#fff5e6,stroke:#ac762c,color:#654516;
+ classDef done fill:#e8f5ef,stroke:#46826a,color:#234d3c;
+ class T,X,Y next;
+ class E done;
+```
+
+### 1.4 Example: cache-aware and session-aware routing
+
+These policies replace steps **2–3** above. `cache_aware` serves plain and
+prefill groups only; decode groups use load-aware (`power_of_two`) unless they set
+`session_aware`. Both policies split the group's engines in two:
+
+- **Preferred engine (A):** for `cache_aware`, the best-ranked prefix owner that
+ passes admission; for `session_aware`, the engine bound to the session.
+- **Normal engines:** the rest of the engine group. When needed, the router picks
+ one of them (**B**) with the load-aware `power_of_two` policy.
+
+A and B are checked against the same group admission limits (for example
+`max_kv_usage` or `max_waiting_requests`; see [configuration examples](#2-configuration-examples)).
+
+- **`prefer` mode (default):** use A if it passes admission.
+- **`balanced` mode:** also pick a B engine and compare the loads of A and B.
+ - Switch to B only when it is clearly less loaded:
+ `load(A) > load_factor × load(B)` **and** `load(A) − load(B) > load_gap`
+ (defaults: factor 2; gap 1024 tokens, or 4 requests). Otherwise, including
+ when either load is unknown, keep A.
+ - `balanced_by` sets the load metric:
+ - **`prefill_tokens` (default):** the engine's reported waiting uncached tokens,
+ plus the tokens this request would prefill on it. For `cache_aware` that
+ subtracts the engine's cached prefix, so A's cache hit counts in its favor.
+ - **`running_requests`:** the engine's reported running requests.
+- **No A, or A fails admission (either mode):** fall back to B.
+
+## 2. Configuration examples
+
+**CLI flags** set the defaults for every engine group. A **bucket file**
+(`--bucket-config`) defines the buckets, and each group in it can override the
+policy, admission, and affinity settings for itself. Without a bucket file,
+there are no length tiers: every request can go to any engine.
+
+Settings fall into these categories:
+
+| Category | What it controls | CLI flags | Bucket-file group key |
+| --- | --- | --- | --- |
+| **Engines** | Plain or P/D, and where engines come from | `--worker-urls`, or `--service-discovery` with `--selector` (plain) or `--prefill-selector` + `--decode-selector` (P/D) | `worker_services`, `worker_ids` |
+| **Policy** | How the router scores engines within a group | `--policy` | `policy` |
+| **Affinity** | When the preferred engine (prefix owner or session binding) wins over a normal engine | `--affinity-*`, `--cache-affinity-*`, `--session-id-header` | `affinity` |
+| **Admission** | Limits that make an engine reject a request | `--max-*` | `admission` |
+| **Buckets** | Which buckets a request fits and the order they are tried | `--bucket-config` | `max_input_tokens`, `max_context_tokens`, `rank`, ... |
+
+How policies apply to each group:
+
+- **Plain and prefill groups** use `--policy` (default `power_of_two`).
+- **Decode groups** always default to `power_of_two`; under reorg,
+ `--decode-policy` must stay `power_of_two`.
+- **A bucket-file group** can set `"policy"` to `power_of_two` or to the
+ `--policy` value, nothing else. `cache_aware` cannot be used on decode, so
+ only `session_aware` can replace `power_of_two` there.
+
+### Example A: plain engines, cache-aware
+
+```bash
+sgl-router --model-id qwen3 --chat-routing reorg \
+ --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
+ --policy cache_aware \
+ --affinity-mode prefer --cache-affinity-min-match-ratio 0.2 \
+ --max-waiting-requests 16 --max-in-flight 56
+```
+
+- **Engines:** two plain engines; each handles both prefill and decode.
+- **Policy:** every engine is scored with `cache_aware`.
+- **Affinity:**
+ - An engine owns the prefix if it holds at least 1024 matched tokens (the
+ default) **and** 20% of the prompt.
+ - `prefer` sends the request to the best owner that passes admission.
+- **Admission:** an engine is rejected when it reports 16 or more waiting
+ requests, or has 56 or more requests in flight from this router.
+
+### Example B: P/D engines, session-aware
+
+```bash
+sgl-router --model-id qwen3 --chat-routing reorg \
+ --service-discovery --service-discovery-namespace inference \
+ --prefill-selector app=qwen3,role=prefill \
+ --decode-selector app=qwen3,role=decode \
+ --policy session_aware --session-id-header x-session-id \
+ --affinity-mode balanced --affinity-balanced-by running-requests \
+ --max-kv-usage 0.9
+```
+
+- **Engines:** Kubernetes discovery finds prefill and decode engines by label.
+- **Policy:**
+ - Prefill engines use `session_aware`: requests with the same
+ `x-session-id` stick to the prefill engine they are bound to.
+ - Decode engines use `power_of_two`. To make decode session-aware too, use a
+ bucket file whose decode group sets `"policy": "session_aware"`.
+- **Affinity:** in balanced mode the router samples an alternative engine, and
+ the session moves to it when the bound engine has more than 2x the
+ alternative's running requests **and** more than 4 more than it (the
+ defaults). The session also moves when the bound engine fails admission.
+- **Admission:** prefill and decode engines are rejected at 90% KV usage.
+
+### Example C: P/D buckets from a file
+
+```bash
+sgl-router --model-id qwen3 --chat-routing reorg \
+ --service-discovery --service-discovery-namespace inference \
+ --prefill-selector app=qwen3,role=prefill \
+ --decode-selector app=qwen3,role=decode \
+ --policy cache_aware \
+ --bucket-config buckets.json
+```
+
+`buckets.json` defines a short tier with a TTFT estimate and a long tier:
+
+```json
+{
+ "ttft_slo": "slo_first",
+ "buckets": [
+ {
+ "id": "short",
+ "max_input_tokens": 4096,
+ "max_context_tokens": 8192,
+ "ttft_ms": 300,
+ "prefill": {
+ "worker_services": ["inference/prefill-short"],
+ "admission": {"max_pending_prefill_tokens": 32768},
+ "affinity": {"mode": "balanced", "load_factor": 1.5}
+ },
+ "decode": {
+ "worker_services": ["inference/decode-short"],
+ "admission": {"max_kv_usage": 0.9}
+ }
+ },
+ {
+ "id": "long",
+ "rank": 1,
+ "max_context_tokens": 131072,
+ "prefill": {
+ "worker_services": ["inference/prefill-long"],
+ "policy": "power_of_two"
+ },
+ "decode": {"worker_services": ["inference/decode-long"]}
+ }
+ ]
+}
+```
+
+- **Engines:** each group takes the discovered engines of its role that belong
+ to the listed Kubernetes Services.
+- **Buckets:**
+ - Requests that fit within 4k input and 8k context fit both buckets; `short`
+ is tried first because it is smaller. Longer requests go only to `long`.
+ - If `short` rejects a request, the router tries `long`.
+- **Policy:**
+ - `short` prefill uses `cache_aware` from `--policy`.
+ - `long` prefill overrides it with `power_of_two`.
+ - Both decode groups use `power_of_two`.
+- **Affinity:** `short` prefill uses balanced mode with a load factor of 1.5.
+ `power_of_two` groups take no `affinity` block; setting one is rejected at
+ startup.
+- **Admission:**
+ - `short` prefill engines reject requests while 32k or more uncached tokens
+ are waiting.
+ - `short` decode engines reject requests at 90% KV usage.
+ - `long` engines set no limits.
diff --git a/experimental/sgl-router/POLICY_DESIGN.md b/experimental/sgl-router/POLICY_DESIGN.md
deleted file mode 100644
index 6c93e00605..0000000000
--- a/experimental/sgl-router/POLICY_DESIGN.md
+++ /dev/null
@@ -1,655 +0,0 @@
-# Engine selection: target design
-
-This document describes the target architecture for engine selection. It defines
-responsibilities and behavior; the interface and configuration examples are
-sketches, not a specification of the current API or CLI. Implementation status is
-listed at the end.
-
-## Principles
-
-1. **Filter buckets by token length, then order preferences.** `BucketResolver`
- returns all compatible buckets, applying optional SLO preferences before
- capacity/rank/ID ordering, without inspecting workers or policies.
-2. **The bucket owns plain versus PD engine selection.** `Bucket::pick_engines`
- calls its one plain group or both prefill and decode groups, returning a
- complete selection. Both PD engines come from that same bucket.
-3. **An engine group owns membership and its policy.** `EngineGroup::pick`
- filters live workers by model, health, stage, and membership, invokes the
- policy, and validates that its result belongs to the exact candidate set.
-4. **A policy owns selection, fallback, admission, and its state dependencies.**
- Construction injects the load, KV, or affinity handles it needs. Request-time
- arguments contain facts and candidates. A policy cannot choose another bucket
- or cross a PD role boundary.
-5. **The handler owns bucket fallback and dispatch.** It calls `pick_engines`
- on each bucket and dispatches only after a complete selection succeeds.
- Missing candidates or admission rejection advance to the next bucket, where
- all required engines are selected again. Invalid signals or policy results stop routing.
-
-An engine is represented by `Worker`. Registry role labels remain authoritative
-when filtering candidates. Groups reference worker IDs rather than owning live
-workers. Group policy instances are reused across requests.
-
-### Data ownership
-
-| Type | Owns |
-| --- | --- |
-| `BucketResolver` | A model's bucket collection, length filtering, and ordering |
-| `Bucket` | ID, length constraints, rank, groups, and complete plain/PD engine selection |
-| `EngineGroup` | Engine membership, attached policy, and engine selection |
-| `WorkerRegistry` | Live workers, model membership, health, and role |
-| Request handler | Request preparation, ordered bucket attempts, HTTP errors, and dispatch |
-
-Omitting both `worker_ids` and `worker_services` includes every healthy engine
-serving the requested model and role. Explicit empty membership lists are rejected
-by the config factory. `EngineGroup::new(policy)` creates a catch-all group.
-`BucketGroups::Pd` requires both groups, making partial or mixed plain/PD bucket
-configurations unrepresentable.
-
-## 1. Code organization
-
-```text
-src/
- buckets.rs Bucket resolution and engine-group selection
- policies/
- mod.rs Policy contract and construction
- admission.rs Per-engine acceptance checks
- cache_aware.rs Cache selection and local/remote adapter
- session_aware.rs Session selection and fallback
- sticky.rs Routing-key selection and fallback
- least_load.rs Least-load selection
- power_of_two.rs Pair sampling and stage-aware comparison
- random.rs Random selection
- round_robin.rs Rotation
- state/
- mod.rs Shared exports
- kv_events/ Local index, subscriptions, hashing, wire format
- load_monitor/ Engine reports and router-local request accounting
- affinity_store.rs Assignments, expiry, and atomic updates
- server/
- app_context.rs Shared service lifecycle and policy wiring
- routes/chat.rs Request preparation, PD coordination, dispatch
-```
-
-During migration, `buckets_reorg.rs` and `policies_reorg/` implement this design
-beside the live `policies/` path. The layout above is the target after switchover.
-Shared state already lives directly under `src/state/`.
-
-Dependencies flow from engine groups to policies, and from policies and admission
-to shared state. State does not depend on bucket ordering or concrete policy
-strategies. Small shared helpers are sufficient; no generic score-composition,
-tier executor, or separate selection framework is required.
-
-## 2. Responsibilities and request flow
-
-`BucketResolver::resolve(input_tokens, expected_peak_tokens, ttft_ms, tokens_per_second)` returns an ordered
-list of compatible bucket references (possibly empty), or an invalid-signal error.
-It does not receive a stage or a load view, resolve live engines, or invoke policies.
-The handler iterates this list until a bucket supplies the complete engine selection.
-
-`Bucket::pick_engines(workers, request)` accepts a `BucketRequest`
-of prepared routing facts, invokes the required groups, and returns `BucketPick`
-(one plain pick or a complete P/D pair). Failures retain their stage. This API
-has no HTTP headers, `AppContext`, or forwarding dependency.
-
-`EngineGroup::pick(workers, request)` resolves healthy workers for the request's
-model and stage, intersects them with its membership, sorts them by stable ID,
-and invokes its attached policy. It rejects foreign results, including a newly
-allocated worker with the same ID as a candidate.
-
-```text
-chat_completions (reorg configured): prepare tokens and expected peak
- |
- v
-BucketResolver::resolve: ordered length-compatible buckets
- |
- v
-For each bucket: bucket.pick_engines(...)
- |
- +-- BucketGroups::Plain
- | plain.pick() -> one plain engine
- |
- +-- BucketGroups::Pd
- prefill.pick() -> P engine
- decode.pick() -> D engine from the same bucket
- |
- +-- empty group / admission rejection -> try next bucket (repeat all picks)
- +-- invalid signal / configuration / foreign pick -> return error
- |
- v
-Complete selection -> forward_chat_request: acquire guards, attach PD bootstrap, forward response
-```
-
-The handler extracts token facts and header keys once into `BucketRequest`.
-The bucket creates a stage-specific `PickRequest` for each group call, supplying
-its own ID and the role associated with that group. Input length, expected peak,
-token IDs, and session/routing keys pass through. Policies obtain observations
-from their own shared-state handles; buckets and handlers do not provide load,
-KV, or affinity services on each call. PD uses separate group policies, but never
-independently resolves a decode bucket. Decode selection
-failure discards that tentative prefill choice and advances to the next bucket
-on missing candidates or admission rejection. No forwarding guards are acquired
-and no prefill request is sent until both picks in one bucket succeed.
-PD compatibility constraints beyond model and role remain follow-up work.
-
-There is one endpoint: `POST /v1/chat/completions`. `AppContext::chat_routing`
-chooses its implementation:
-
-- `ChatRouting::Legacy` (default) uses the existing policies and bucket selector.
-- `ChatRouting::Reorg(HashMap)` uses the new bucket and
- policy interfaces, with explicit model-specific resolvers.
-
-`--chat-routing reorg` constructs default plain/PD buckets using the existing
-`--policy` and tuning flags. Programmatic callers can still install explicit
-resolvers. A missing model returns 404 without falling back to legacy routing.
-The default serving path remains legacy.
-
-Both implementations reuse request preparation (including sampling validation
-and tokenization), forwarding, streaming, middleware, and the 32 MiB body limit.
-The reorg implementation requests tokenization for length matching, retaining
-the existing body-size estimate when tokenization is unavailable.
-
-## 3. Bucket resolution
-
-1. Validate that a known expected peak is at least the input length.
-2. Keep buckets whose inclusive input-token range contains the input length.
-3. Check the bucket context capacity against input plus requested output when
- known, or against input length when the output budget is unknown.
-4. Apply enabled SLO preferences to complete buckets. Count unmet preferences
- equally, then sort by ascending input capacity (the lesser of the input upper
- bound and context capacity), rank, and ID. Unbounded capacities sort last.
- Return the entire ordered list, retaining nonpreferred buckets for fallback.
-5. The handler calls each bucket's `pick_engines` until one supplies its complete
- selection. A failed PD attempt never contributes an engine to a later pair.
-
-The handler checks addition overflow when computing the expected peak.
-An empty bucket list becomes a 400 `NoMatchingBucket` response. After exhausting
-the list, accumulated admission rejection details produce a selection failure
-(503); if there were no admission rejections, the last unavailable stage produces
-a stage-specific 503. Invalid policy signals/configuration or out-of-candidate
-picks stop the pass immediately with an internal error. Successful engine
-selection ends the pass; forwarding errors do not restart bucket iteration.
-
-Token ranges and rank belong to the bucket, not its engine groups. For a PD
-bucket, both groups share this one request-length decision. Policy fallback on
-a cache/affinity miss stays within that group's candidates. There is no second
-pass with relaxed admission and no post-policy substitution.
-
-### Optional SLO ordering
-
-`Bucket` has optional `ttft_ms` and `tokens_per_second` estimates. The resolver
-has independent `ttft_slo` and `tps_slo` preferences: `Disabled` (default),
-`SloFirst` (matching first), and `BestEffort` (nonmatching first). The handler
-parses `x-sgl-ttft-slo-ms` and `x-sgl-tps-slo` only when their preference is enabled;
-invalid enabled headers return 400 before dispatch. Disabled headers are ignored.
-SLO targets belong to bucket resolution, not the engine policy's `PickRequest`.
-
-Absent targets are neutral. A bucket matches TTFT when its positive estimate is
-at most the target, and throughput when its finite positive estimate is at least
-the target. Missing or invalid estimates do not match a supplied target. Enabled
-TTFT targets must be positive; throughput targets must be finite and positive.
-
-Each unmet preference adds one ordering penalty. With both preferences set to
-`SloFirst`, a bucket matching both comes before one matching either, followed by
-buckets matching neither. Capacity/rank/ID breaks ties within these tiers. The
-same logic applies to plain and PD buckets, and a PD bucket always supplies both
-engines. TTFT and throughput preferences never independently resolve P/D groups.
-Length constraints are applied first and admission rejection still advances to
-the next complete bucket, including a bucket outside the preferred SLO tier.
-
-Global session modes and sticky policies are follow-ups. Their
-integration must preserve bucket-first selection and the same-bucket PD rule.
-Cross-bucket affinity probing is not part of this interface. Session/routing
-keys still pass through `PickRequest` for policies operating inside the selected
-group; unsupported legacy modes need explicit migration decisions before the
-standard serving path switches.
-
-## 4. Policy and admission contracts
-
-### Policy
-
-All policies implement one asynchronous, object-safe `Policy::pick` interface.
-A boxed future supports the remote prefix indexer; local policies can return an
-immediately ready result.
-
-```rust
-pub trait Policy: Send + Sync + std::fmt::Debug {
- fn pick<'a>(
- &'a self,
- engines: &'a [Arc],
- request: &'a PickRequest<'a>,
- ) -> BoxFuture<'a, Result>;
-}
-```
-
-Policies own their required state handles and read observations locally. Nested
-fallback uses `pick_fallback(engines, request)`, which calls the fallback's `pick`;
-the fallback reads its own state. There is no shared observation context or cache.
-Buckets and HTTP handlers supply only candidates and request facts.
-
-`Pick` identifies one engine and a selection reason for metrics and tracing.
-`PickRequest` carries model, stage, selected bucket ID, input and optional
-expected peak counts, optional token IDs, and session/routing keys. It contains
-no HTTP body, bucket resolver, state handles, snapshots, or backend configuration.
-
-| Outcome | Meaning |
-| --- | --- |
-| `Pick` | The chosen engine belongs to the supplied set and passed admission |
-| `NoMatchingBucket` | No bucket supports the requested length |
-| `NoCandidates` | No eligible member or policy selection miss |
-| `NoAdmissibleEngine` | A policy exhausted its candidates; includes rejection reasons |
-| `AdmissionRejected` | The chosen engine failed after-selection admission |
-| `InvalidSignal` / `InvalidConfiguration` | Invalid policy input or configuration |
-| `OutsideCandidates` | Policy returned an engine outside its exact candidate set |
-
-### Admission
-
-Admission evaluates acceptance. It does not rank engines, choose replacements,
-change buckets, or mutate affinity.
-
-An admission policy is a set of per-engine caps, `AdmissionLimits`. Each cap
-is optional; an unset cap is not checked, and the default admits everything.
-Each cap admits while the engine's load is below it. Usage caps are shares, in
-(0, 1], of the engine's reported capacity, so one limit fits engines of any
-size. The request itself is not counted: an engine just below a cap still takes
-it, and the engine queues or retracts if it overflows. Admission observes load
-without reserving it.
-
-| Limit | Checks |
-| --- | --- |
-| `max_running_usage` | Running requests against reported max running requests |
-| `max_kv_usage` | KV tokens against reported KV capacity |
-| `max_waiting_requests` | Reported waiting requests |
-| `max_pending_prefill_tokens` | Reported waiting uncached tokens |
-| `max_inflight_requests` | Router-local in-flight requests |
-
-```json
-{"max_running_usage": 0.9, "max_kv_usage": 0.95, "max_inflight_requests": 64}
-```
-
-Unknown fields are rejected during deserialization.
-
-The policy reads the selected engine's `EngineMetrics` from the load snapshot
-it already captured for selection plus the live in-flight counter and calls
-`EngineAdmission::check(engine, metrics)`, which returns `Allow`,
-`Reject(limit name)`, or an error. Reported
-metrics are `None` without a fresh, complete report, never zero, and such
-limits fail open; the in-flight count is always known. Policies attach the
-checker as `Arc` and decide where checking belongs in
-their selection algorithm; there is no placement setting or filtering wrapper.
-
-Power-of-two first selects an engine, then calls admission exactly once on that
-engine. A rejection returns `AdmissionRejected` to the bucket loop; it does not
-resample, choose the other sampled engine, or run a policy fallback. No candidates
-returns `NoCandidates` without invoking admission. A single candidate is selected
-directly; otherwise two distinct candidates are sampled uniformly, and the one
-with lower stage pressure wins. A complete tie keeps the first sampled engine.
-
-Power-of-two reuses the existing pure pressure-comparison functions. Plain and
-prefill stages compare estimated prefill queue time when both reports provide it,
-then waiting uncached tokens, waiting requests, and running requests. Decode
-compares waiting requests, running requests, KV usage fraction, then used KV tokens.
-Reported-pressure ties use router-local active requests. If either sampled engine
-lacks a fresh, complete native report with valid capacity, both are compared by
-router-local active requests instead. Basic reports from older publishers are
-still passed to admission when fresh, but do not supply native pressure metrics.
-
-Power-of-two retains the selected engine's load record from selection and
-passes it to admission without another snapshot. A single candidate still has
-its load read for admission, even though selection needs no comparison.
-Neither the bucket nor HTTP handler supplies observations. Synchronous checks
-do not fetch telemetry over the network themselves.
-
-`AdmissionLimits::default()` is the default for new explicit policy attachments.
-It leaves health, role, membership, and policy preferences in force. Migrated
-configurations must retain their existing capacity and configured budget checks;
-see compatibility below.
-
-Affinity preferences never bypass attached hard admission.
-
-Admission checks observe capacity; they do not reserve it. Concurrent requests
-may pass against the same observation. Strict reservations would require a
-separate mechanism.
-
-## 5. Concrete policies
-
-| Policy | Selection behavior |
-| --- | --- |
-| `RoundRobinPolicy` | Rotate over candidates using a cursor owned by this policy instance |
-| `RandomPolicy` | Choose uniformly from candidates |
-| `PowerOfTwoPolicy` | Sample two distinct candidates when possible and choose the lower-pressure engine using the stage's load comparison |
-| `LeastLoadPolicy` (`load_based`) | Choose the least loaded engine; preserve tie-breaking, telemetry fallback, and recent-dispatch correction |
-| `SessionAwarePolicy` | Prefer the session binding; use admitted power-of-two fallback and rebind |
-| `StickyPolicy` | Reuse an admitted routing-key binding; use the configured fallback for new or missing keys |
-| `CacheAwarePolicy` | Prefer an admitted prefix owner; use power-of-two fallback |
-
-Session assignments are scoped by model, bucket ID, stage, and session key.
-`SessionAwarePolicy::new(store, engine_load)` receives shared state; the caller
-owns the store's idle timeout and eviction task. Missing or empty session keys
-use power-of-two without creating assignments. A new or out-of-group binding
-uses power-of-two with `AdmissionLimits::default()`, then the session policy
-checks its selected engine before binding. A concurrent live assignment wins,
-but is checked before returning it; rejection ends that attempt without
-rewriting the binding or retrying another engine. Existing bindings follow the
-shared affinity modes below. Session policies can be attached
-independently to each role. Programmatic reorg callers configure
-`model.affinity.session_id_header` for HTTP header extraction; this does not
-enable legacy global modes or backup escape.
-
-Session and sticky policies do not create assignments for missing keys. A
-binding outside the candidates cannot win. A missing binding may invoke policy
-fallback within the group. Rejected session affinity tries fallback without the
-old engine; a successful admitted replacement rebinds the session. Failed fallback
-preserves the old binding and advances to the next bucket.
-
-Both affinity policies support `--affinity-mode prefer` (default) and `balanced`.
-Prefer retains admissible affinity. Balanced compares it with an admitted
-power-of-two alternative and switches only when the `--affinity-balanced-by` load
-exceeds both the alternative times `--affinity-load-factor` (default 2) and the
-alternative plus `--affinity-load-gap`. The load is `prefill-tokens` (default;
-gap default 1024): native waiting uncached tokens plus the request's uncached
-tokens on that engine, so the alternative also pays for the prefix the cache owner
-holds. Session-aware has no prefix signal and counts the whole input on both
-engines. `running-requests` (native or basic reports, gap default 4) ignores the
-request. Both reports must be fresh; missing data
-or ties preserve affinity. Bindings commit during selection after
-admission, as with initial placement; dispatch failure does not roll them back.
-
-Sticky fallback supports `round_robin`, `random`, `power_of_two`, and `load_based`,
-with round-robin as the default. Nested fallbacks use
-`AdmissionLimits::default()`; the owning policy explicitly checks the engine
-returned by its fallback.
-
-### Cache-aware behavior
-
-The architecture preserves the cache algorithm within the supplied candidate
-set. Its responsibilities are:
-
-1. Look up prefix ownership through the local index or remote indexer.
-2. Apply minimum matched-token and optional ratio thresholds.
-3. Bound candidates using prefix/pressure ordering and the configured minimum,
- ratio, and maximum worker counts.
-4. Check admission and choose the best remaining prefix holder.
-5. Apply the shared affinity mode. With no admitted prefix owner, fall back
- within the group, excluding rejected engines.
-
-Fallback samples two engines and checks the selected engine's admission; it
-does not resample on rejection. Candidate limits apply only to prefix selection.
-
-Memoize the prefix lookup once per request, including remote I/O. Each policy
-restricts those matches to its own candidates. A memoized lookup does not imply
-that another bucket or stage's cache selection and admission have already run.
-Any optimization that skips those steps must establish that the previous result
-applies to the current group.
-
-## 6. Shared state and construction
-
-Application wiring starts shared services once. Policy construction validates
-configuration and passes the required handles to each policy and admission
-implementation. For example, `PowerOfTwoPolicy::new(Arc)`
-retains the application's shared load table. KV-aware and affinity-aware policies
-receive their corresponding shared handles when implemented. Policies with no
-state dependency require none. Policy instances do not create duplicate
-subscriptions, polling loops, indexes, or remote-client concurrency limits.
-
-Requirements come from all configured policies, their admission checks, and
-nested fallbacks. This includes tokenization, affinity-header extraction, load
-observations, and dispatch timestamps. A role-group override must receive its
-required settings even when the model's default uses a different policy.
-
-### Load state
-
-`state/load_monitor/` owns engine reports and existing router-local request
-accounting. Power-of-two owns an `Arc` and captures a snapshot
-locally for each nonempty selection attempt. Its `pick` method selects the engine,
-then passes that engine's borrowed load record directly to admission.
-No snapshot or observation is added to `Pick`, `PickRequest`,
-or the bucket interface, and no shared observation context is threaded through
-policies or fallbacks.
-
-The existing snapshot reader preserves rank aggregation, freshness, and capacity
-fields. Missing, stale, or rank-incomplete reports yield `None`, not zero load.
-A new pick reads current state. Admission reuses the selected observation even if
-reports change after selection; it neither recaptures nor reserves capacity.
-Fallback policies read their own state and do not share snapshots with callers.
-
-Snapshot capture still scans the full table; an engine-scoped reader can be added
-if profiling justifies it. Power-of-two ranks engines with the shared prefill/decode
-comparisons in `state/load_monitor/engine_ranking.rs`, including router-local
-fallback. Concrete load-aware admission remains
-in #40271. Further shared load interpretation and correction for dispatches since
-the report remain follow-ups; these must preserve source, freshness, and available
-measurements without adding another
-independent in-flight counter. Load and cache observations are not an atomic global
-snapshot. Preserve request-guard cleanup.
-
-### Cache state
-
-`state/kv_events/` owns local subscriptions, event application, hashing, and the
-radix-tree index. The tree stores prefix ownership and storage tiers, not KV
-tensors. Eviction, invalidation, and worker removal update this shared state.
-
-A small `CacheSource` adapter beside the cache policy normalizes local and remote
-lookups into common prefix results. Preserve model/cache namespaces, required
-rank information, and hash/block-size handling. Convert block counts to token
-counts only with a valid conversion.
-
-A confirmed miss and an unavailable backend both allow cache fallback, but
-remain distinguishable in diagnostics. Invalid queries and configuration errors
-remain explicit errors. Remote indexing and load-only deployments do not need
-a duplicate local cache tree.
-
-### Affinity state
-
-`AffinityStore` owns scoped assignments, idle expiry, worker invalidation, and
-atomic updates. Policies decide when to reuse or replace an assignment.
-
-Concurrent first assignments must converge on an effective binding that is
-still a candidate and passes admission. If a concurrent binding returns a
-different engine, the policy must check that engine before returning it.
-Reconcile conflicts with bounded retry.
-
-Create or replace a binding only after admission succeeds. A binding records
-preferred placement, not successful execution, so it may remain if later PD
-selection or dispatch fails. It must not increment dispatch accounting.
-
-## 7. Configuration and compatibility
-
-Under `--chat-routing reorg`, `--bucket-config` reads a JSON file of complete
-buckets. Each bucket sets `plain`, or both `prefill` and `decode`; each group may
-set `worker_ids` or `worker_services`, plus `policy`, `admission` and `affinity`.
-`affinity` overrides `mode`, `balanced_by`, `load_factor` and `load_gap` for a
-session- or cache-aware group; unset fields take the CLI values.
-
-```json
-{
- "ttft_slo": "slo_first",
- "buckets": [
- {
- "id": "short-context",
- "max_input_tokens": 4096,
- "max_context_tokens": 8192,
- "ttft_ms": 300,
- "prefill": {"worker_services": ["inference/prefill-short"], "admission": {"max_pending_prefill_tokens": 32768}},
- "decode": {"worker_services": ["inference/decode-short"], "admission": {"max_kv_usage": 0.9}}
- },
- {
- "id": "long-context",
- "rank": 1,
- "max_context_tokens": 131072,
- "prefill": {"worker_services": ["inference/prefill-long"]},
- "decode": {"worker_services": ["inference/decode-long"]}
- }
- ]
-}
-```
-
-A request with 4k input tokens and a 16k expected peak cannot fit the short
-bucket's context capacity. It selects the long bucket and both of its P/D groups.
-With a known peak of 8k or less, the same input selects both groups of the short bucket.
-
-Membership has two mutually exclusive forms. `worker_ids` pins exact discovery
-identities: URLs for static discovery, `namespace/pod-UID` for Kubernetes pods.
-`worker_services` lists Kubernetes `namespace/service` names and matches any
-listed Service through its EndpointSlices' `kubernetes.io/service-name` label.
-Service membership survives pod replacement and scale-up; worker IDs still
-change per pod incarnation so breaker, load and affinity state cannot carry over.
-Only Services included by the discovery watch are visible. Multiple slices and
-Services selecting one pod contribute the union of its memberships; removing a
-slice removes only memberships no longer advertised by another slice. Membership
-updates preserve the live worker and its in-flight state. Readiness and routing
-use the same group filter, including model, health and role constraints.
-
-Omitted group fields mean every engine of the group's role and `--policy`
-(power-of-two on decode). Admission limits a group leaves unset take
-`--max-in-flight` / `--max-kv-usage`, field by field. JSON `null` also inherits;
-there is no per-group disable value. To limit only some groups, leave the CLI
-default unset and configure those groups explicitly. A group policy is
-`power_of_two` or the `--policy` kind, whose affinity, cache and tokenizer
-settings are the ones resolved. Startup rejects empty or duplicate IDs, a bucket
-that is not exactly plain or P/D, inverted token ranges, a minimum input above
-the context capacity, empty membership lists, blank worker IDs, malformed
-Service names, both membership fields together, non-positive capacity or SLO
-estimates, usages outside (0, 1], zero count limits, and a policy a stage cannot
-serve. Without the flag, reorg builds one plain and one P/D bucket
-over all engines. The worker registry still rejects mixed plain and PD engines
-within one model.
-
-Legacy `--bucket-config` files describe role-specific buckets and do not
-translate: P/D associations must be written explicitly, never inferred from
-rank or names.
-
-Retained settings keep their meanings, defaults, units, and validation unless a
-change is listed below. Policy-specific tuning applies to role groups using that
-policy. Reject unsupported settings and incompatible combinations at startup;
-do not accept and ignore them.
-
-### Retained behavior
-
-- Keep `load_based` as the CLI name for `LeastLoadPolicy`.
-- Preserve explicit engine membership and context constraints. Bucket-level
- ranges and ordering replace independent per-stage selection.
- SLO preferences order whole buckets; deployment configuration remains explicit
- until the production configuration factory and serving switchover are ready.
-- Preserve cache-provider selection, endpoint validation, query timeout and
- concurrency limits, and unavailable-backend fallback.
-- Preserve the 1,024-token default minimum cache hit, optional ratio gate,
- and candidate bounds. Shared affinity modes replace cache diversion knobs.
-- Preserve session and sticky headers, idle timeouts, eviction cadence, and the
- four sticky fallback choices. Global modes need a bucket-first migration design.
-- Map `--filter overloaded` and `--max-in-flight` to `max_inflight_requests`;
- the existing router-local counter remains the source. `--max-kv-usage` sets
- `max_kv_usage` for groups that leave it unset.
-- Preserve configured capacity, pending-prefill, and in-flight checks, including
- their missing-report behavior. Power-of-two applies admission to its selected
- engine; other policies explicitly place checks in their selection logic.
- Other paths use `AdmissionLimits::default()` unless a check is configured.
- Never silently discard a configured budget.
-
-Listener and shutdown configuration, discovery, worker health and circuit
-breakers, tokenizer loading, request timeouts, sampling overrides, and logging
-remain outside the policy redesign. Preserve their existing behavior and
-validation, including dispatch-time breaker probes and request cancellation.
-
-### Deliberate changes
-
-| Behavior | Target |
-| --- | --- |
-| Power-of-two admission | Check only the chosen engine; rejection advances to the next bucket |
-| Round-robin cursor | One cursor per role-group policy instance |
-| Capacity exhaustion | Try the next compatible bucket; return accumulated rejection details if all fail |
-| Primary/backup proposals and post-policy substitution | Removed; each policy returns one engine |
-| Session affinity | Reuse or replace bindings through shared affinity modes |
-| Omitted `--affinity-mode` | Defaults to `prefer` for both affinity policies |
-| Pressure-guard tuning | Replaced by `--affinity-load-factor` and `--affinity-load-gap` |
-| Policy attachment | Explicit role-group policy overrides the applicable model/stage default |
-
-Reject these dropped options explicitly:
-
-- `--policy fused_score` and `--policy score_policy`, including `--fuse` terms
- and weights.
-- `--decode-policy legacy_host_affinity`.
-- `--stable-pair`.
-- `--affinity-mode soft` / `strict`; use `prefer` / `balanced`.
-- Legacy pressure guards, cache switch margins, queue limits and saturation floors.
-- `--filter prefix_cache` and `--prefix-cache-min-share`. The removed prefix-share
- filter is not equivalent to the cache-aware minimum-hit gate.
-
-## 8. Dispatch and failure handling
-
-A successful pick means admission passed against the observed state. Health and
-capacity can change before dispatch. The handler owns network operations, retry
-rules, and accounting for the engines actually dispatched to.
-
-Selection fallback advances through the ordered compatible buckets before any
-network dispatch. Every PD attempt picks both engines from that bucket. With
-`--retry-max-attempts` above 1, a dispatch that fails before any response
-reaches the client reruns selection from the first bucket with the failed
-engine excluded (`BucketRequest::excluded`); it never replaces a pick whose
-response the client has started to receive.
-
-For PD, acquire and release accounting for the actual stages and clean up
-partial setup on failure. Policy selection does not own the PD request lifetime.
-Selection metrics must not imply that dispatch or execution succeeded.
-
-## Implementation status
-
-This PR adds the side-by-side interfaces in `src/buckets_reorg.rs` and
-`src/policies_reorg/`, plus a configurable bucket-first implementation behind
-`chat_completions`. The live `src/policies/` path remains the default.
-
-Implemented here:
-
-- `BucketResolver::resolve` returns all length-compatible buckets in optional
- SLO-preference tiers, with capacity/rank/ID order within each tier.
-- `Bucket::pick_engines` owns plain/PD orchestration and stage-specific policy
- requests; `BucketRequest` carries prepared facts and `BucketPick` retains picks.
-- `Bucket` owns input limits, context capacity, rank, and plain-or-PD groups.
-- `EngineGroup::pick` owns live candidate filtering, policy invocation, and
- exact candidate validation, without cross-bucket fallback.
-- `Policy::pick`, within-group fallback interface, per-engine `EngineAdmission::check`,
- and `AdmissionLimits` with running and KV usage shares plus waiting,
- pending-prefill and in-flight counts. Power-of-two samples two distinct engines, compares stage pressure,
- and checks its selected engine with no replacement on rejection.
-- Policy-owned load dependency and local observations. Power-of-two passes the
- selected engine's load record directly to admission, without another snapshot.
- `PickRequest`, `Pick`, and bucket APIs carry no load observations.
-- The reorg chat implementation iterates resolved buckets, calls `pick_engines`,
- advances on empty candidates/admission rejection, and
- dispatches only after one complete selection. Exhaustion retains admission reasons.
-- `AppContext::chat_routing` configures legacy versus reorg routing on the same
- endpoint and carries the reorg model-resolver map.
-- Optional bucket TTFT/throughput estimates and preferences, with enabled-header
- parsing and whole-bucket fallback in the shared chat route.
-- `CacheAwarePolicy` reads local radix-tree or remote indexer prefixes, intersects
- exact worker URLs with the current group, applies hit thresholds and candidate
- bounds, and applies the shared affinity mode.
-- `PrefixMemo` shares lookup results (including misses and unavailable backends)
- across bucket attempts for one prepared request. Entries are keyed by the shared
- `Arc` so different index namespaces remain independent. Each pick
- reruns its own candidate filtering and admission after obtaining a fresh snapshot.
-- Cache selection checks bounded candidates explicitly; rejected owners are
- excluded from fallback. The cache policy checks its fallback winner's admission.
- Cache policies require plain/prefill groups.
-- `SessionAwarePolicy` reuses admitted model/bucket/role-scoped bindings from a
- shared `AffinityStore`, falling back to power-of-two for new or keyless sessions.
- Assignments follow admission; concurrent binding winners are rechecked.
- Rejected affinity invokes fallback; failed fallback preserves the binding.
- The caller owns expiry and sweeper lifecycle. A binding may remain after a
- later PD group fails, because it records placement rather than dispatch.
-
-- `--bucket-config` builds explicit plain and P/D buckets with per-group
- membership, policy and admission; omitting it builds the default buckets.
-
-Follow-up work includes remaining selection policies.
-
-Not yet implemented in the reorg path:
-
-- Other concrete selection policies.
-- Global session modes and sticky routing-key affinity.
-- Power-of-k cache-miss fallback configuration and cache decision metrics.
-- Shared load interpretation, dispatch correction, and policy-specific
- dispatch-timestamp requirements.
-- PD compatibility filtering and legacy-route switchover.
-
-The preceding policy sections describe target behavior for those follow-ups;
-they do not claim those capabilities are present in this PR.
diff --git a/experimental/sgl-router/README.md b/experimental/sgl-router/README.md
index ccb1305e09..50ed7b9839 100644
--- a/experimental/sgl-router/README.md
+++ b/experimental/sgl-router/README.md
@@ -1,738 +1,353 @@
# sgl-router
-Slim, KV-aware, OpenAI-compatible router for SGLang workers.
+Slim, KV-aware router for SGLang workers.
-Serves a single model and routes across its workers. Exposes
-`/v1/tokenize`, `/v1/detokenize`, `/v1/models`, [`/v1/embeddings`](#embeddings),
-[`/v1/classify`](#classify), [`/v1/rerank`](#rerank),
-[`/v1/chat/completions` and `/v1/completions`](#openai-over-generate) and SGLang's native
-[`/generate`](#native-generate) (buffered and SSE), plus
-`/healthz` / `/readyz` and `/metrics`. Worker pools come from either a static URL
-list or Kubernetes EndpointSlice discovery. Both edges speak cleartext HTTP/2
-where the peer does — see [HTTP/2](#http2).
+A router serves one model and spreads its requests across that model's SGLang
+engines. It picks an engine per request from live engine load, global KV-cache prefix
+matches, or session affinity, supports prefill/decode (P/D) disaggregation, bucketed deployments, and
+discovers engines from a static URL list or Kubernetes EndpointSlices.
-## Building
+- [Architecture](#architecture)
+- [Quick start](#quick-start)
+- [Key interfaces](#key-interfaces)
+- [Routing policies](#routing-policies)
+- [Deployment examples](#deployment-examples)
+- [Add a new policy](#add-a-new-policy)
+- [Other features](#other-features)
+
+Related docs: [ARCHITECTURE.md](ARCHITECTURE.md) (routing design and bucket
+configuration), [REFERENCE.md](REFERENCE.md) (detailed feature behavior),
+[monitoring/README.md](monitoring/README.md) (metrics and
+Grafana), [BENCHMARKS.md](BENCHMARKS.md),
+[sgl-kv-indexer/README.md](sgl-kv-indexer/README.md) (external KV indexer).
+
+## Architecture
+
+A deployment is a **router** in front of **buckets of SGLang engines**. A bucket
+is a logical group defined in the router configuration. It holds either one
+**plain** engine group, where each engine does prefill and decode, or separate
+**prefill and decode (P/D)** groups. Buckets can be length tiers (for example
+short and long context); without a bucket file there is one plain bucket and one
+P/D bucket containing every discovered engine.
+
+
+
+Engines publish load reports and KV events. The router uses them, plus its own
+in-flight counts and session bindings, to pick engines. An optional
+[external KV indexer](sgl-kv-indexer/README.md) can replace the router's local
+radix tree as the source of prefix matches. In P/D, the router sends the request
+to both the chosen prefill and decode engines, and prefill transfers KV directly
+to decode.
+
+For each request, the router renders and tokenizes it, tries the buckets that fit
+its length (ordered by SLO preference, capacity, then `rank`), lets the group's
+policy pick an engine that passes admission, and dispatches it, retrying on
+another engine if it fails before any response. Output handling (OpenAI
+responses, tool and reasoning parsing) also happens in the router.
+
+**Scalability.** Routers are stateless: nothing is persisted or shared between
+replicas. Each one rebuilds its routing state (engine load, KV prefix tree,
+in-flight counts) from the engines' own reports, so you scale a large deployment
+by running more router replicas behind a load balancer. A new replica can copy
+the KV tree from a warm sibling at startup
+([peer bootstrap](#kv-cache-signals)). Session bindings stay in each
+replica's memory, so with `session_aware` and several replicas, route a session's
+requests to the same replica (for example, a load balancer sticky on the session
+header).
+
+## Quick start
+
+Build:
```bash
cd experimental/sgl-router
cargo build --release
```
-## Running
-
-The router is configured entirely through CLI flags (run
-`sgl-router --help` for the full list). It serves exactly one model, so
-`--model-id` is required, along with exactly one discovery backend.
-`--tokenizer-path` is optional: give it a local `tokenizer.json` path or a
-HuggingFace repo id, and when omitted the router downloads the tokenizer
-for `--model-id` from HuggingFace (honoring `HF_TOKEN` / `HF_HOME`).
-
-Static worker list:
+Run in front of two engines with cache-aware routing:
```bash
-sgl-router \
+./target/release/sgl-router \
--host 0.0.0.0 --port 30000 \
--model-id qwen3 \
- --tokenizer-path /models/qwen3/tokenizer.json \
- --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000
-```
-
-Kubernetes EndpointSlice discovery:
-
-```bash
-sgl-router \
- --host 0.0.0.0 --port 30000 \
- --model-id qwen3 \
- --tokenizer-path /models/qwen3/tokenizer.json \
- --service-discovery \
- --service-discovery-namespace prod \
- --selector app=engines-qwen3
-```
-
-Omit `--service-discovery-namespace` to watch all namespaces (requires
-cluster-wide RBAC). For prefill/decode disaggregation, replace `--selector`
-with `--prefill-selector` and `--decode-selector`.
-
-To run two engine versions side by side in PD mode (e.g. during a rollout),
-pass `--pd-version-group-label `. A prefill worker is then paired only with
-decode workers that have the same value for that EndpointSlice label (inherited
-from the Service, so use one Service per role and version), so KV never crosses
-versions. Unlabeled workers form their own group.
-
-```bash
-sgl-router --model-id qwen3 --service-discovery \
- --prefill-selector app=engines-qwen3,role=prefill \
- --decode-selector app=engines-qwen3,role=decode \
- --pd-version-group-label sglang.ai/version-group
-```
-
-External KV indexer as the cache-aware signal source:
-
-```bash
-sgl-router \
- --model-id qwen3 \
- --tokenizer-path /models/qwen3/tokenizer.json \
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
- --policy cache_aware \
- --cache-prefix-provider indexer \
- --kv-indexer-endpoint http://10.0.0.10:50051 \
- --kv-indexer-query-timeout-ms 100 \
- --kv-indexer-query-max-inflight 32
-```
-
-The Indexer replaces the Router-local radix tree as the native Cache-Aware
-signal. Query timeouts and local concurrency are bounded by the two Indexer
-options, which default to 100 ms and 32 respectively.
-
-### LoRA cache identity
-
-LoRA routing requires workers that publish `BlockStored.lora_name`. Requests
-select the adapter through `model: "base:adapter"` on OpenAI routes, or
-`lora_path` on native routes. Salted generation requests also carry `cache_salt`.
-
-Worker block IDs are opaque and may differ across adapter loads or workers.
-The router derives adapter-aware lookup keys from event tokens and keeps a
-per-worker, per-DP-rank mapping for parent links and removals. Multiple block
-IDs can own the same lookup key; removing one leaves the other owners and
-storage tiers intact. The remote indexer bridge performs the same translation.
-Base-model events retain their existing published-key path.
-
-New replicas request `?include_lora=true` snapshots containing LoRA mappings
-(version 2). Legacy consumers receive a base-only version-1 view, so mixed
-snapshots do not prevent base-cache bootstrap during rolling upgrades. Base-only
-snapshots retain version 1 for every consumer. LoRA stores buffered during bootstrap retain their
-tokens; the original base-model buffering and replay behavior is unchanged.
-
-Reloading different adapter weights under the same public name can still
-credit stale engine blocks until their eviction. Names identify adapters;
-versioned names are needed to distinguish different weights.
-
-### KV-event gap replay
-
-ZMQ drops KV events at the publisher's high-water mark, and a lost removal
-leaves the router crediting a worker with blocks it no longer holds. Give each
-engine a replay socket and the router re-fetches the batches a sequence gap
-skipped:
-
-```bash
-python -m sglang.launch_server ... \
- --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558"}'
-```
-
-Rank r publishes on 5557 + r and replays on 5558 + r, so with `dp_size > 1`
-place the replay base at least dp_size ports past the PUB base. The engine keeps
-the last `buffer_steps` batches (default 10000); older gaps stay unrepaired.
-`sgl_router_kv_event_replays_total{outcome}` counts the attempts.
-
-### Pending prefixes
-
-Until the engine's KV events arrive, requests that share a cold prefix see no
-cached owner and spread across workers, each prefilling the same prefix. This
-is common with parallel sampling, RL rollouts and agent fan-out.
-`--cache-pending-prefix-ttl-ms` credits a worker with a prompt's prefix for that
-long after routing it there, so the burst stays together. It is off by default
-and needs `--policy cache_aware` with the Router-local radix tree.
-`sgl_router_cache_pending_prefix_hits_total` counts lookups where a pending
-prefix matched deeper than any confirmed one; the policy's candidate and
-admission filters still decide whether that worker is picked.
-
-### Peer bootstrap (Kubernetes)
-
-A replica that starts mid-fleet subscribes to each worker's KV topic
-mid-stream, so everything already resident in the engines' caches is invisible
-to it and it routes cache-blind until traffic re-stores those blocks —
-degrading the engines' locality for the whole fleet, once per replica on every
-rolling update. With a peer selector set, a booting replica instead pulls a
-tree snapshot from a warm sibling over `/internal/kv_snapshot` and splices it
-under its own live delta stream, and `/readyz` stays 503 until that bootstrap
-settles. Off unless `--kv-peer-selector` is set; requires `--policy
-cache_aware` with the Router-local radix tree (not an external Indexer) and
-`--service-discovery`, which the CLI enforces.
-
-```bash
-sgl-router \
- --model-id qwen3 --policy cache_aware \
- --service-discovery --service-discovery-namespace prod \
- --selector app=engines-qwen3 \
- --kv-peer-selector kubernetes.io/service-name=sgl-router
-```
-
-- `--kv-peer-selector` is matched against **EndpointSlice labels** — the
- router Service's labels plus `kubernetes.io/service-name`, never the pods'
- labels, so `kubernetes.io/service-name=` is the unambiguous
- choice. A pod-template label matches nothing and every replica boots cold.
-- `--kv-bootstrap-timeout-ms` (default 600000) bounds the whole bootstrap;
- readiness waits on it, so startup/readiness probes must tolerate a replica
- staying unready this long.
-- `--kv-bootstrap-fetch-timeout-cap-ms` (default 300000) caps one snapshot
- fetch within that budget.
-- `--kv-bootstrap-seed-required` (off by default) keeps holding `/readyz` when
- siblings were present but their tree could not be pulled, bounded at
- max(3× the bootstrap timeout, 60s) — it delays a failed seed's replica (and
- with it a rolling update) rather than shipping it cache-blind; it cannot
- stall a rollout indefinitely.
-
-The deployment needs two things beyond the flags: the router's ServiceAccount
-must hold `get`/`list`/`watch` on `endpointslices` in the watched namespace
-(the worker-discovery Role already grants this), and the pod spec should wire
-`POD_NAME`, `POD_NAMESPACE` and `POD_IP` from the downward API so a replica
-can exclude itself from its own peer list. A rolling update should surge
-(`maxUnavailable: 0`) so new replicas always have warm siblings to copy from.
-`tests/e2e/k8s_integration/manifests/kv-bootstrap.yaml` is a worked example of
-all of it, and the `sgl_router_kv_bootstrap_*` series in
-[monitoring/README.md](monitoring/README.md) make the whole path observable.
-
-### Reorg routing
-
-Use `--chat-routing reorg` to select the new bucket engine. The existing `--policy`
-and cache/session flags configure its policies; a bucket file is optional.
-
-```bash
-sgl-router --model-id qwen3 --worker-urls http://localhost:30001 \
--chat-routing reorg --policy cache_aware
```
-Reorg supports `power_of_two` (its default), `cache_aware`, and `session_aware`.
-Discovery supplies the plain or PD workers; decode uses power-of-two. Cache
-settings, external indexers and session headers/timeouts retain their existing
-flags. Admission flags set the default per-engine caps; an engine at a cap is
-skipped, and affinity falls back to the others:
+Send a request:
+
+```bash
+curl http://localhost:30000/v1/chat/completions \
+ -H 'Content-Type: application/json' \
+ -d '{"model": "qwen3", "messages": [{"role": "user", "content": "Hello"}]}'
+```
+
+`--model-id` is required. `--tokenizer-path` takes a local `tokenizer.json` or a
+HuggingFace repo id; when omitted, the tokenizer for `--model-id` is downloaded
+from HuggingFace (honoring `HF_TOKEN` / `HF_HOME`). `sgl-router --help` lists
+every flag.
+
+## Key interfaces
+
+### HTTP endpoints
+
+| Endpoint | Purpose |
+| --- | --- |
+| `POST /v1/chat/completions`, `POST /v1/completions` | OpenAI API, served [over `/generate`](#openai-over-generate) |
+| `POST`/`PUT /generate` | SGLang [native generate](#native-generate), buffered or SSE |
+| `POST /v1/messages`, `/v1/messages/count_tokens` | [Anthropic Messages](#anthropic-messages) |
+| `POST /v1/embeddings`, `/v1/classify`, `/v1/rerank` | [Embeddings, classify and rerank](#embeddings-classify-and-rerank) |
+| `POST /v1/tokenize`, `/v1/detokenize` | Tokenize with the router's tokenizer |
+| `GET /v1/models` | The served model |
+| `POST /flush_cache` | Fan out a cache flush to every worker |
+| `GET /healthz`, `/readyz` | Liveness and readiness |
+| `GET /metrics` | Prometheus metrics, see [monitoring/README.md](monitoring/README.md) |
+
+Both edges speak cleartext HTTP/2 where the peer does; see
+[operations](#operations). With `--grpc-port`, the router also serves the
+engine's [native gRPC](#native-grpc) API.
+
+### Key flags
+
+| Area | Flags |
+| --- | --- |
+| Model | `--model-id`, `--tokenizer-path`, `--no-tokenizer` |
+| Engines | `--worker-urls`, or `--service-discovery` with `--selector` (plain) or `--prefill-selector` + `--decode-selector` (P/D) |
+| Routing | `--chat-routing reorg`, `--policy`, `--bucket-config`, `--affinity-*`, `--session-id-header` |
+| Admission | `--max-in-flight`, `--max-waiting-requests`, `--max-pending-prefill-tokens`, `--max-running-usage`, `--max-kv-usage` |
+| Reliability | `--retry-max-attempts`, `--request-timeout-secs`, `--stale-request-timeout-secs` |
+| KV signals | `--cache-prefix-provider`, `--kv-indexer-endpoint`, `--kv-peer-selector` |
+
+Omitting `--chat-routing` keeps the legacy routing path (default
+`--policy round_robin`). This README and [ARCHITECTURE.md](ARCHITECTURE.md)
+describe `--chat-routing reorg`.
+
+## Routing policies
+
+Each engine group runs one policy, chosen by `--policy` (default `power_of_two`
+under reorg) or per group in a bucket file:
+
+- **`power_of_two`**: samples two engines and takes the less loaded. Prefill and
+ plain engines compare estimated prefill queue time, waiting uncached tokens,
+ waiting, running, then in-flight requests; decode engines compare waiting and
+ running requests, then KV usage. Decode groups always default to it.
+- **`cache_aware`**: prefers the engine holding the longest cached prefix of the
+ prompt (at least `--cache-affinity-min-matched-tokens`, default 1024, and
+ optionally `--cache-affinity-min-match-ratio` of the prompt). Plain and
+ prefill groups only. Needs a tokenizer.
+- **`session_aware`**: keeps requests with the same session header
+ (`--session-id-header`, default `x-session-id`) on the engine they are bound
+ to. Any stage, including decode via a bucket file.
+
+Both affinity policies fall back to `power_of_two` among the remaining engines
+when there is no preferred engine or it fails admission. With
+`--affinity-mode prefer` (default) the preferred engine wins whenever it is
+admitted. With `--affinity-mode balanced` the router also samples an
+alternative and switches only when the preferred engine's load exceeds both
+`alternative * --affinity-load-factor` (default 2) and
+`alternative + --affinity-load-gap`, measured by `--affinity-balanced-by`
+(`prefill-tokens` by default, or `running-requests`).
+
+Admission flags set per-engine caps; an engine at a cap is skipped:
| Flag | Rejects an engine at |
|---|---|
-| `--max-in-flight N` | `N` requests this router has in flight to it (no `--filter overloaded` needed) |
+| `--max-in-flight N` | `N` requests this router has in flight to it |
| `--max-waiting-requests N` | `N` engine-reported waiting requests |
| `--max-pending-prefill-tokens N` | `N` engine-reported waiting uncached tokens |
| `--max-running-usage F` | share `F` of its reported running-request capacity |
| `--max-kv-usage F` | share `F` of its reported KV capacity |
-Engine-reported caps lag by one load report, so a burst can pass them; the
-router-local `--max-in-flight` counts dispatches as they happen.
-Unsupported legacy options fail at startup.
+Engine-reported caps lag by one load report, so a burst can pass them;
+`--max-in-flight` counts dispatches as they happen.
-`--bucket-config buckets.json` replaces the default plain and P/D buckets. Each
-bucket is plain or P/D, and each group may set its own engines, policy,
-admission and affinity; see [POLICY_DESIGN.md](POLICY_DESIGN.md#7-configuration-and-compatibility):
+For the full selection rules see ARCHITECTURE.md:
+[load-aware routing](ARCHITECTURE.md#13-example-load-aware-routing),
+[cache- and session-aware routing](ARCHITECTURE.md#14-example-cache-aware-and-session-aware-routing),
+and [configuration examples](ARCHITECTURE.md#2-configuration-examples).
-```json
-{"buckets": [{
- "id": "default",
- "prefill": {"worker_services": ["inference/prefill"], "admission": {"max_pending_prefill_tokens": 32768}},
- "decode": {"worker_services": ["inference/decode"], "admission": {"max_kv_usage": 0.9}}
-}]}
-```
+## Deployment examples
-`worker_services` matches Kubernetes Services by `namespace/name`, using the
-`kubernetes.io/service-name` label on watched EndpointSlices. Replacement pods
-and new replicas join the same group automatically. The router's discovery
-selectors must include those EndpointSlices; this field does not expand the watch.
-A worker selected by several Services belongs to each of them.
-
-For static URL discovery, use `"worker_ids": ["http://worker:30000"]`: each ID
-is the configured worker URL. Kubernetes worker IDs are `namespace/pod-UID`
-(and change when a pod is replaced), so use `worker_services` for durable pools.
-Set only one membership field, or omit both for every engine of the group's role.
-Empty membership lists and blank worker IDs are rejected at startup.
-
-Admission fields left unset or set to `null` inherit CLI defaults. To apply a
-limit only to selected groups, omit that CLI default and set it on those groups.
-
-Omitting `--chat-routing` keeps the existing policies and defaults.
-
-Both reorg affinity policies accept `--affinity-mode prefer` (default) or
-`balanced`. Prefer keeps an admissible session binding or the best admissible
-prefix owner. Balanced samples a power-of-two alternative and switches only
-when the affinity engine's load exceeds both
-`alternative * --affinity-load-factor` (default 2) and
-`alternative + --affinity-load-gap`. `--affinity-balanced-by` picks the load:
-`prefill-tokens` (default; gap default 1024) is the engine's waiting uncached
-tokens plus the prompt tokens it would prefill for this request, so a cache owner
-is credited for its prefix (session-aware assumes the whole prompt on either
-engine); `running-requests` (gap default 4) ignores the request. Missing fresh load preserves admissible
-affinity; ties also preserve it.
-
-A bucket group may override these with `"affinity": {"mode", "balanced_by",
-"load_factor", "load_gap"}`; unset fields take the CLI values, and the balanced
-fields require `"mode": "balanced"`. For example, a session-aware group balanced
-by running requests:
-`"plain": {"affinity": {"mode": "balanced", "balanced_by": "running_requests"}}`.
-
-Both modes fall back within the group when affinity fails admission, excluding
-rejected engines. The fallback winner must pass admission; failure advances to
-the next bucket. Session replacements are bound after admission during selection,
-not after dispatch. Reorg rejects legacy pressure guards, cache switch margins,
-and queue/saturation gates in favor of these shared affinity settings.
-
-### Optional tokenizer for load-only routing
-
-`--no-tokenizer` skips tokenizer loading for load-only policies such as
-`power_of_two` and `session_aware`, on either routing path. Workers tokenize the
-original messages, and `/v1/tokenize` and `/v1/detokenize` are unavailable.
-Cache-aware routing, prefix-cache terms or filters, and buckets with token-length
-or context limits require a tokenizer. Reorg buckets that only select membership
-and load-based policies can use `--no-tokenizer`.
-
-### DP-rank routing
-
-An engine launched with `--dp-size` or `--attn-dp-size` runs several DP ranks,
-each with its own KV cache, behind one endpoint. With `--dp-aware`, the router
-also picks the rank inside the selected worker. It sends that rank as
-`X-Data-Parallel-Rank`, which the engine honors, and overwrites any value the
-client sent. The router picks the first of these that applies:
-
-1. A hash of the sticky routing key or session id, so a conversation keeps
- its rank and router replicas agree.
-2. The rank with the deepest cached prefix in the local KV tree.
-3. The rank with the fewest requests this router has in flight on it, or with
- `--dp-rank-policy round_robin`, the worker's ranks in turn.
-
-In PD mode, decode is ranked by step 3 only. The decode request names the
-prefill rank in `disagg_prefill_dp_rank`, and the bootstrap room satisfies
-`room % prefill_dp_size == prefill_rank`.
-
-### Retries
-
-`--retry-max-attempts N` (default 1, which disables retries; 3 is typical) lets
-a request that fails before any response reaches the client be sent again to a
-worker it has not tried yet. A failure is a transport error, an open circuit
-breaker, a 5xx, a 429, or a timeout: `--request-timeout-secs` bounds a whole
-non-streaming response, and a streaming one until its headers arrive. SGLang's
-chat endpoint sends those headers with the first token, so for streaming chat
-the same flag bounds time to first token, queueing included; raise it for
-long-context or deeply queued fleets. Once a streaming response's 2xx status
-has been sent, it is never retried, and neither is any other 2xx or 4xx. In PD mode the failed
-side is excluded, or both sides when the caller set `rid`, which an engine still
-running it would refuse; a new pair gets a new bootstrap room. Retries share the
-request's `--stale-request-timeout-secs` deadline, and none starts after it. When
-every eligible worker has failed, the client gets the last failure.
-`sgl_router_retries_total` counts the retried attempts.
-
-Each retry first waits out a jittered exponential backoff: a random delay in
-`[d/2, d]`, where `d` starts at `--retry-initial-backoff-ms` (default 50),
-doubles per retry, and is capped at `--retry-max-backoff-ms` (default 2000).
-Set the initial backoff to 0 to retry immediately. A backoff never waits past
-the request's stale deadline.
-
-### Engines with `--api-key`
-
-The router reads each worker's `/server_info` and `/model_info` to learn its
-model, KV-event publisher, HTTP/2 support and DP size. An engine launched with
-`--api-key` rejects those requests without the key, so pass the same key as
-`--worker-api-key`. Chat requests and `/flush_cache` forward the caller's
-`Authorization` header instead, so callers still need the engine key.
-
-### Fleet-wide sampling contract
-
-`--override-sampling-params` fixes the sampling configuration for every client
-of this router, independently of what the engine's own defaults happen to be
-(on native [`/generate`](#native-generate), in each prompt's `sampling_params`):
+Static workers, plain engines:
```bash
-sgl-router \
- --model-id qwen3 \
- --tokenizer-path /models/qwen3/tokenizer.json \
- --worker-urls http://10.0.0.1:30000 \
- --override-sampling-params '{"temperature": 1, "top_p": 0.95, "n": 1}' \
- --sampling-param-conflict reject
+sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
+ --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000
```
-It takes one JSON object keyed by the request-body field names
-(`temperature`, `top_p`, `top_k`, `min_p`, `repetition_penalty`,
-`frequency_penalty`, `presence_penalty`, `n`). A configured value is injected
-whenever the request omits that field, so the engine's defaults cannot drift
-from what the operator declared. `temperature`, `top_p`, `top_k`, `min_p` and
-`repetition_penalty` are the five the engine resolves from the model's own
-`generation_config`, which is what makes them drift when a deployed image
-changes; the rest have fixed API defaults.
+Kubernetes discovery, plain engines (omit the namespace to watch all
+namespaces, which needs cluster-wide RBAC):
-An explicit `null` counts as omitting the field, not as a client-supplied
-value: the OpenAI API types these parameters as nullable with a documented
-default, so `null` asks for the default — and on a governed fleet the
-configured value is what the default is.
+```bash
+sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
+ --service-discovery --service-discovery-namespace prod \
+ --selector app=engines-qwen3
+```
-For a request that does send a value, `--sampling-param-conflict` decides:
-`reject` (the default) 400s a differing value before admission, while `allow`
-forwards the client's value untouched — the router never silently rewrites what
-a client sent. A `reject` response carries
-`x-router-error-code: sampling_contract_violation` and is counted in
-`sgl_router_sampling_contract_rejections_total{param}`, so a rollout's blast
-radius is visible per parameter rather than folded into `bad_request`.
+Kubernetes discovery, P/D engines with session affinity:
-`reject` also 400s a value it cannot read as a number, on a governed parameter
-only. The engine coerces more than JSON numbers — a bool and a numeric string
-both become numbers — by rules that are undocumented and need not match across
-a fleet, so a value the router cannot read is one it cannot prove conforms, and
-waving it through would make the pin bypassable. The common coercions are
-matched exactly (`false` is 0, `"0.5"` and `"1_0"` are 0.5 and 10), so this
-refuses only genuine garbage. `allow` is unaffected: it promises nothing, so
-such a value keeps flowing to the engine, which owns the request schema.
+```bash
+sgl-router --model-id qwen3 --chat-routing reorg --policy session_aware \
+ --service-discovery --service-discovery-namespace prod \
+ --prefill-selector app=engines-qwen3,role=prefill \
+ --decode-selector app=engines-qwen3,role=decode \
+ --max-kv-usage 0.9
+```
-A value may also be an inclusive band, `{"min": LO, "max": HI}`, for a
-parameter that stays tunable inside a range. A band names no value to inject,
-so it constrains only the requests that name the parameter; one that omits it
-gets the model's own `generation_config` default, which the router cannot see.
-Because a band can only ever reject, combining one with `allow` is a startup
-error.
+Length-tiered buckets from a file (schema and a full example in
+[ARCHITECTURE.md](ARCHITECTURE.md#example-c-pd-buckets-from-a-file)):
-Values are range-checked at startup, so a misconfiguration fails the launch
-instead of 400ing every request at the engine. Repeating a key in the flag is
-also a startup error, rather than silently enforcing whichever copy came last.
+```bash
+sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
+ --service-discovery --service-discovery-namespace prod \
+ --prefill-selector app=engines-qwen3,role=prefill \
+ --decode-selector app=engines-qwen3,role=decode \
+ --bucket-config buckets.json
+```
-| parameter | accepted | notes |
-| --- | --- | --- |
-| `temperature` | `[0, 2]` | |
-| `top_p` | `(0, 1]` | |
-| `top_k` | `>= 1`, or exactly `-1` | `-1` is the engine's "whole vocabulary" spelling and its default. Being non-contiguous it cannot bound a band. Note `top_k: 1` is greedy decoding, not "disabled". |
-| `min_p` | `[0, 1]` | not an OpenAI parameter; the engine's domain |
-| `repetition_penalty` | `(0, 2]` | not an OpenAI parameter; the engine's domain |
-| `frequency_penalty` | `[-2, 2]` | |
-| `presence_penalty` | `[-2, 2]` | |
-| `n` | `[1, 128]` | |
+## Add a new policy
-The OpenAI domains are deliberately narrower than what the engine itself
-accepts (`SamplingParams.verify` would take `temperature: 5`): these values are
-injected into request bodies, and a fleet contract outside the range every
-OpenAI client library validates against is far more likely a typo than an
-intent.
+Reorg policies live in [`src/policies_reorg/`](src/policies_reorg/). Contributions
+are welcome; read [ARCHITECTURE.md](ARCHITECTURE.md) first for how buckets,
+admission and fallback fit together.
-Cost: a request that named every configured field forwards its original bytes
-untouched. One that omits a field has the scalars spliced directly into the
-body bytes — no parse, no re-serialize — which on a 16 MiB body is ~0.24 ms
-against ~5.7 ms for a `serde_json` round-trip. Only `input_ids` and PD
-bootstrap injection still parse and re-serialize, because those may have to
-overwrite a key the client sent.
+1. **Implement the `Policy` trait** in `src/policies_reorg/.rs`. The trait
+ is in [`src/policies_reorg/mod.rs`](src/policies_reorg/mod.rs):
-### Relationship to the engine's own `--preferred-sampling-params`
+ ```rust
+ pub trait Policy: Send + Sync + Debug {
+ fn pick<'a>(&'a self, engines: &'a [Arc], request: &'a PickRequest<'a>)
+ -> BoxFuture<'a, Result>;
+ fn supports(&self, _stage: Stage) -> bool { true }
+ fn needs_request_tokens(&self) -> bool { false }
+ fn fallback(&self) -> Option<&dyn Policy> { None }
+ }
+ ```
-The engine has an inject-when-absent flag of its own,
-`--preferred-sampling-params`, merged in
-`python/sglang/srt/managers/tokenizer_manager.py` as
-`{**preferred, **obj.sampling_params}`. On `/v1/chat/completions` it is
-currently a no-op: `ChatCompletionRequest.to_sampling_params`
-(`python/sglang/srt/entrypoints/openai/protocol.py`) resolves every sampling
-key eagerly through `generation_config` and then its own defaults, so the
-right-hand side of that merge is always fully populated and always wins.
-(`/v1/responses`, in the same file, already omits `None` entries for exactly
-this reason.) If that is fixed engine-side, `--preferred-sampling-params`
-covers the inject-when-absent half for a single engine.
+ - `pick` must return one of the supplied `engines` and check admission on it
+ (`self.admission.check(&engine, &metrics)`), returning
+ `PickError::AdmissionRejected` on rejection.
+ - The policy holds its own shared-state handles (load table, KV source,
+ session store), passed in at construction, and reads them inside `pick`.
+ - Override `supports` to restrict stages (plain, prefill, decode),
+ `needs_request_tokens` if it reads `request.token_ids` or `request.prefix`,
+ and `fallback` if it delegates to another policy.
+ - [`power_of_two.rs`](src/policies_reorg/power_of_two.rs) is a complete
+ ~80-line example; [`session_aware.rs`](src/policies_reorg/session_aware.rs)
+ shows affinity with a fallback.
+2. **Declare the module** in `src/policies_reorg/mod.rs`.
+3. **Add a `PolicyKind` variant** in [`src/config/types.rs`](src/config/types.rs)
+ with its `--policy` name. `PolicyKind` is shared with the legacy path, so add
+ an arm returning an error to `build_kind` in
+ [`src/policies/factory.rs`](src/policies/factory.rs) (the compiler lists every
+ exhaustive match to update).
+4. **Wire it into** [`src/policies_reorg/factory.rs`](src/policies_reorg/factory.rs):
+ add it to the allowlist in `validate()` and construct it in the `match kind`
+ of `Groups::build`.
+5. **Test it** in `tests/component/policies_reorg_.rs`, registered in
+ `tests/component/main.rs`, following `policies_reorg_power_of_two.rs`. For
+ end-to-end routing, add a case under `tests/proxy/chat_routing/reorg/`.
+6. **Document it**: a subsection in [ARCHITECTURE.md](ARCHITECTURE.md) next to
+ section 1.4, and a bullet in [Routing policies](#routing-policies) above.
-What stays the router's to own either way is the enforcement half: the
-`reject` immutability contract with its 400 before admission — costing no
-queue slot and no engine round-trip — the `sampling_contract_violation` code
-and per-parameter counter, bands, and one contract applied at a shared ingress
-across engines whose own flags the router operator may not control.
+## Other features
-## Chat rendering
+Each item links to its full description in [REFERENCE.md](REFERENCE.md).
-The router renders chat requests with dynamo-render (`dynamo-renderer`): the model's
-HF Jinja template from `tokenizer_config.json` or a sibling
-`chat_template.jinja`, or dynamo-render's built-in DeepSeek encoder (V4 family, V3.2)
-for template-less models. Cache-aware routing hashes the rendered tokens so its
-prefix queries match the blocks the engine caches. Models the engine encodes in
-code but dynamo-render cannot tokenize here (Inkling) route via raw prompt
-text, as does any model whose template fails to load or render.
+### Native `/generate`
-Some chats additionally forward the rendered tokens to the engine as
-`input_ids`, retaining the original messages, so the engine skips
-re-tokenizing. How many depends on the model's renderer. DeepSeek-V4's, V4.1's
-and Kimi-K3's native encoders and GLM-5.3's and Qwen3.8's templates are
-fixture-verified against SGLang's request normalization, so they forward every
-chat except multimodal ones and those with caller-provided `input_ids`. Renderers
-without that verification (HF Jinja templates)
-forward only plain text chat requests (string `content`, no tools, no template
-kwargs or reasoning controls or historical `reasoning_content`, no assistant
-continuation, no consecutive users or non-leading system turns) and warn
-`UNVERIFIED` at startup; every other request shape is rendered for routing
-only.
+Same body and response as the engine's `/generate`, buffered or SSE. With a
+tokenizer loaded, `text` is forwarded as `input_ids` so routing sees the
+engine's exact tokens (`--disable-input-ids-forwarding` turns this off). See
+[native `/generate`](REFERENCE.md#native-generate).
-Use matching model files on the router and workers, and set
-the same `--default-chat-template-kwargs`, `SGLANG_DEFAULT_THINKING`,
-`SGLANG_DSV4_REASONING_EFFORT`, and `SGLANG_DSV41_REASONING_EFFORT` on both.
-The router reads these render defaults from its own configuration and environment;
-it does not discover the workers' settings. Point `--tokenizer-path` at the workers'
-model snapshot so the V4 effort profile is read from the same
-`encoding/encoding_dsv4.py`.
+### OpenAI over /generate
-Set `--disable-input-ids-forwarding` for this router's model when worker-side
-rendering has not been verified to match. This disables router-generated IDs
-for every routing policy; cache-aware routing still renders and tokenizes
-locally, and the original messages reach the workers for engine processing.
-Caller-supplied `input_ids` remain caller-owned and pass through unchanged.
+For `/v1/chat/completions` and `/v1/completions`, the router runs SGLang's
+OpenAI layer itself: it sends the request to `/generate` and builds the OpenAI
+response. Requests it cannot reproduce exactly (for example media or strict
+tools) go to the engine's own route unchanged. See
+[OpenAI over /generate](REFERENCE.md#openai-over-generate) for the exact
+conditions, and [chat rendering](REFERENCE.md#chat-rendering) plus the
+per-model sections ([DeepSeek V4 and V4.1](REFERENCE.md#deepseek-v4-and-v41),
+[GLM-5.3](REFERENCE.md#glm-53), [Qwen3.8](REFERENCE.md#qwen38),
+[Kimi-K3](REFERENCE.md#kimi-k3)) for which chats forward `input_ids`.
-Forwarding logs its assumptions at startup. In particular, disable it when the
-router's render defaults differ from the workers', for worker template overrides
-not reflected in the router's model files, for worker parser overrides such as
-`--tool-call-parser deepseekv32` that select a native encoder over a shipped
-template, or conversation templates with stop strings (the engine's `input_ids`
-path skips those template stops). Disabling forwarding preserves engine behavior
-but does not establish parity for local routing hashes.
+### Anthropic Messages
-Also set `--disable-input-ids-forwarding` for array-only templates: Dynamo may wrap
-string content into arrays differently from the worker. The pinned Dynamo renderer does not expose
-its conversion flag, so the router cannot automatically block these templates.
-Detailed content-format parity coverage follows in #39133.
+`/v1/messages` is converted to a chat completion and back.
+`/v1/messages/count_tokens` is answered by the router from the chat template.
+See [Anthropic Messages](REFERENCE.md#anthropic-messages).
-The Dynamo crates are pinned exactly and `Cargo.lock` is committed; CI builds
-with `--locked`, so rendered bytes cannot change without a reviewed diff.
+### Embeddings, classify and rerank
-Router tokenization sits on the TTFT path for every chat it renders. Two opt-in
-flags make it cheaper:
+Engine request and response bodies, routed on load. Not available on P/D fleets.
+See [embeddings](REFERENCE.md#embeddings), [classify](REFERENCE.md#classify)
+and [rerank](REFERENCE.md#rerank).
-- `--tokenizer-backend fast` encodes with fastokens (decoding stays on HF). It
- needs a `tokenizer.json` and falls back to `hf` when fastokens cannot load it.
-- `--tokenizer-l1-cache-mb N` caches prefix tokenizations at special-token
- boundaries, so a multi-turn chat encodes only the turns added since the
- previous request. Boundaries are unconditional, non-normalized, non-stripping
- special tokens with no overlapping added-token spellings. Unsafe candidates
- are excluded; if none remain, encoding proceeds without the cache.
+### Native gRPC
-On a ~69K-token DeepSeek-V4 chat, `hf` encodes in ~40 ms, `fast` in ~4 ms, and
-a new turn on a cached history in ~0.2 ms. The DeepSeek fixtures check every
-case under `hf`, `fast`, and `fast` with L1. Startup logs report the resolved
-backend and cache state. `/metrics` exposes only
-`sgl_router_tokenizer_l1_tokens_total{source="cached"|"encoded"}` to measure
-how much tokenization work the cache reuses.
+`--grpc-port` serves the engine's `sglang.runtime.v1.SglangService`. Each RPC is
+routed like its HTTP route and forwarded to the same RPC on the engine's own
+native gRPC port, so engines must run with `--grpc-port` too. See
+[native gRPC](REFERENCE.md#native-grpc).
-## Native `/generate`
+### Bucket file
-`/generate` (`POST` or `PUT`) has the engine's interface: the same
-`GenerateReqInput` body and the same response, buffered or SSE. The body names no
-model, so requests go to the one this router serves. Worker selection
-(`--chat-routing`, `--policy`), PD dispatch, and abort-on-disconnect are shared
-with chat completions.
+`--bucket-config` defines buckets whose groups set `worker_services`
+(`namespace/name`) or `worker_ids`, plus `policy`, `admission` and `affinity`;
+unset fields inherit the CLI flags. See
+[ARCHITECTURE.md](ARCHITECTURE.md#2-configuration-examples) and
+[bucket file membership](REFERENCE.md#bucket-file-membership).
-With a tokenizer loaded, the router tokenizes `text` (a string or a list) with
-the special tokens SGLang adds (BOS per `add_bos_token` for Llama-, Gemma- and
-Cohere-class tokenizers, otherwise the `tokenizer.json` post-processor's), and
-forwards it as `input_ids`. The engine skips tokenizing and routing sees its exact
-tokens. Multimodal requests keep `text`, since the engine expands placeholders
-from it, and `--disable-input-ids-forwarding` keeps it for every request. So does
-a model whose `tokenizer.json` normalizer transformers replaces on load (legacy
-SentencePiece Llama files, bge-m3), or whose `tokenizer_config.json` the router
-cannot read, as the router cannot reproduce its tokens. A batch goes to one
-worker: load counts every prompt and each of its `n` samples, while bucket and
-context limits bound the longest prompt plus its own `max_new_tokens`.
+### KV cache signals
-Otherwise the body passes through, plus PD bootstrap fields, a minted `rid` for
-a single prompt that has none, and `--override-sampling-params` defaults. Under
-`--dp-aware` each worker's body also carries the chosen `routed_dp_rank`; a PD
-batch or `n > 1` request leaves the prefill rank to the engine, which gives item
-`i` the bootstrap room `room + i`.
+- **External KV indexer**: `--cache-prefix-provider indexer
+ --kv-indexer-endpoint `; see [sgl-kv-indexer](sgl-kv-indexer/README.md)
+ and [external KV indexer](REFERENCE.md#external-kv-indexer).
+- **KV-event gap replay**: add `"replay_endpoint"` to the engine's
+ `--kv-events-config` so the router can re-fetch dropped KV events; see
+ [gap replay](REFERENCE.md#kv-event-gap-replay).
+- **Pending prefixes**: `--cache-pending-prefix-ttl-ms` keeps bursts that share
+ a cold prefix on one worker until its KV events arrive; see
+ [pending prefixes](REFERENCE.md#pending-prefixes).
+- **Peer bootstrap (Kubernetes)**: `--kv-peer-selector
+ kubernetes.io/service-name=` makes a new replica copy the KV
+ tree from a sibling before `/readyz` passes; see
+ [peer bootstrap](REFERENCE.md#peer-bootstrap-kubernetes).
+- **LoRA cache identity**: with workers that publish `BlockStored.lora_name`,
+ prefix matches are kept per adapter; see
+ [LoRA cache identity](REFERENCE.md#lora-cache-identity).
-## OpenAI over /generate
+### Operations
-`/v1/completions` and `/v1/chat/completions` behave as the engine's own routes,
-but the router runs SGLang's OpenAI layer itself (`sglang-processor`'s `openai` module): it lowers
-the request to the `GenerateReqInput` SGLang would build, sends it to
-`/generate` like a native request, and builds the OpenAI response, buffered or
-SSE, from the engine's output. The engine's `/server_info` supplies the server
-args that layer reads (`--enable-cache-report`,
-`--stream-response-default-include-usage`, `--incremental-streaming-output`, the
-custom-labels header). Requests it cannot reproduce exactly go to the engine's
-own route as sent: workers that disagree on those args or never
-reported them, `--completion-template`, `return_hidden_states` and other
-`sglext` outputs, `echo` or `logprobs` without a router tokenizer, and bodies
-SGLang would reject. Chat also needs a renderer verified against SGLang (the
-`AllText` scope: DeepSeek-V4, V4.1, GLM-5.3, Kimi-K3 and Qwen3.8 today), router `--default-chat-template-kwargs`
-equal to the workers', a reasoning parser the processor reproduces
-(`deepseek-v4`, `deepseek-v41`, `glm45`, `kimi_k3`, `qwen3`) or none, and no media yet. Tools are served with `tool_choice`
-`auto` or `none`, non-strict, with standard JSON-schema types, and a tool
-parser the processor reproduces (`deepseekv4`, `deepseekv41`, `kimi_k3`, `qwen3_coder`); `required`, named and strict
-tools take a constraint from the engine's xgrammar, so they go to the engine
-(`kimi_k3`'s `auto` tag is reproduced).
-With `--tool-call-parser glm47`, a chat with tools also goes to the engine: its
-grammar needs `ebnf_full_assistant`, which `/generate` does not accept.
-The router reads the model's `config.json` and `generation_config.json` beside
-`--tokenizer-path` (or from the HF repo), as the engine reads them, so a local
-path should point into the model's directory. Engine env vars that change the
-layer (`SGLANG_TOOL_STRICT_LEVEL`, `SGLANG_FORWARD_UNKNOWN_TOOLS`) are taken as
-unset. `sgl_router_openai_route_total` counts each outcome.
-
-## Embeddings
-
-`/v1/embeddings` has the engine's interface: the same OpenAI `EmbeddingRequest`
-body and response. As for chat completions, `model` must name the served model.
-A PD fleet answers 400, since prefill and decode engines serve no embeddings.
-
-Text `input` (a string or a list) is tokenized and forwarded as token IDs, as for
-[`/generate`](#native-generate), plus the EOS SGLang appends for EmbeddingGemma.
-Blank prompts and multimodal items stay as sent, for the engine to reject or
-render. A list is a batch for one worker, routed on load. A single prompt with no
-`rid` gets a minted one. Under `--dp-aware` the engine picks the rank, since its
-embeddings endpoint reads none.
-
-## Classify
-
-`/v1/classify` has the engine's interface: the same `ClassifyRequest` body and
-response. It is served as embeddings are, except that a batch of text stays text,
-since the engine takes token IDs for only one prompt.
-
-## Completions
-
-`/v1/completions` has the engine's interface and is prepared like chat (sampling
-contract, PD bootstrap, DP rank), with the prompt forwarded as sent: tokens the
-router computes for routing do not replace it.
-
-## Rerank
-
-`/v1/rerank` has the engine's interface: the same `V1RerankReqInput` body and
-response, sent as `POST` or `PUT` to the model this router serves. The body is
-forwarded as sent, since the engine renders and tokenizes each query-document
-pair. Routing is on load, with each pair's size estimated from its text. As for
-embeddings, a PD fleet answers 400 and `--dp-aware` pins no rank.
-
-## Anthropic Messages
-
-`/v1/messages` converts each request to a chat completion at ingress, so it is
-routed exactly like `/v1/chat/completions`, and converts the reply (or stream)
-back to the Messages format. Consecutive turns of the same role are merged, as
-the Messages API does, and a final assistant turn is sent as a prefill
-(`continue_final_message`). Besides the Messages fields, only
-`chat_template_kwargs` and `continue_final_message` reach the chat request.
-Every error uses the Anthropic error envelope; 5xx messages are replaced with a
-generic one, and a failed stream still closes its open block and ends with
-`message_stop`.
-
-`/v1/messages/count_tokens` is answered by the router from the model's chat
-template, without calling an engine. Image tokens are not counted, and a model
-without a chat template answers 500 rather than a raw-text estimate.
-
-`thinking` sets the `thinking` / `enable_thinking` chat template kwargs. The
-engine's own `/v1/messages` also adapts it per reasoning parser (for example
-`reasoning_effort` for Hunyuan), which the router cannot see; on such models
-use `output_config.effort` or `chat_template_kwargs` directly.
-
-## DeepSeek V4 and V4.1
-
-Native V4 and V4.1 rendering comes from `rust/sglang-processor`, shared with SGLang's Rust
-server, and follows SGLang's serving path (`serving_chat.py`), not Dynamo's
-OpenAI defaults: all declared tools are rendered with SGLang's schema
-defaults, reasoning effort comes from `reasoning` / `reasoning_effort`, and the
-official/preview effort profile is detected from the checkpoint's
-`encoding/encoding_dsv4.py` or overridden by `dsv4_reasoning_effort_profile` in
-`config.json`, as in SGLang. Reference prompts live in `tests/fixtures/deepseek/`
-and are regenerated by `tests/scripts/generate_deepseek_parity.py`.
-
-V4.1 Flash uses Dynamo's V4.1 encoder with SGLang's numeric reasoning budgets
-and tool payloads. Developer messages and media are left to the worker, since
-the encoder renders them differently.
-
-## GLM-5.3
-
-GLM-5.3 and GLM-5.3-Flash render with the checkpoint's `chat_template.jinja` through
-`rust/sglang-processor`, which builds the template context SGLang's Jinja path builds,
-so every text chat forwards `input_ids`. Media, a `chat_template` kwarg, `-0` in tools
-or tool arguments, and U+001C..U+001F in assistant history render on the worker. With
-the cookbook's parsers, chats without tools are served through `/generate`; chats with
-tools keep the engine's route (see above).
-
-## Qwen3.8
-
-Qwen3.8 (Max, 27B and Flash-Next) renders with the checkpoint's `chat_template.jinja`
-through `rust/sglang-processor`, which picks it by its template and builds the
-template context SGLang's Jinja path builds. Every text chat forwards `input_ids`.
-Media, a `chat_template` key in `chat_template_kwargs`, and text or tool numbers
-Python's `trim` or `json.dumps` treat unlike Dynamo's (such as `1e-06`) render on the
-worker. With the cookbook's `qwen3` and `qwen3_coder` parsers, text chats are served
-through `/generate`; thinking follows the template's `enable_thinking` (on by default).
-
-## Kimi-K3
-
-Kimi-K3 renders through `rust/sglang-processor` (dynamo-render's XTML formatter
-with SGLang's request semantics) and is tokenized with the checkpoint's chunked
-tiktoken encoding; shapes the formatter renders differently go to the worker.
-`--tokenizer-path` accepts a
-local `tiktoken.model` or an HF repo id, whose `tiktoken.model`, `config.json`
-and `tokenizer_config.json` are downloaded when it has no `tokenizer.json`.
-With the cookbook's `kimi_k3` parsers, text chats are served through `/generate`.
-
-## HTTP/2
-
-There is nothing to configure. The router negotiates per connection inbound and
-resolves the protocol per worker outbound; every combination below is reached
-automatically, and HTTP/1.1 remains a supported peer on both edges.
-
-**Inbound.** The listener accepts cleartext HTTP/2 (h2c, prior knowledge) and
-HTTP/1.1 on the same `--port`, chosen per connection. A mesh sidecar or
-load balancer that prefers h2c multiplexes over one connection instead of
-opening one per request; an HTTP/1.1 client is unaffected.
-
-**Outbound.** At registration the router reads each worker's `/server_info` and
-forwards over cleartext h2c only when that worker reports `--enable-http2` (the
-engine's Granian server, which serves h2c and HTTP/1.1 together) **and** the
-worker URL is cleartext. Everything else uses the default client, which speaks
-HTTP/1.1 in cleartext and negotiates ALPN `h2, http/1.1` over TLS:
-
-| `/server_info` | worker URL | router forwards over |
-|---|---|---|
-| `enable_http2: true` | `http://` | cleartext h2c |
-| `enable_http2: true` | `https://` | HTTP/2 over TLS, by ALPN |
-| `enable_http2: false`, or absent | any | HTTP/1.1 (cleartext) or ALPN (TLS) |
-
-The choice is per worker, not fleet-wide, so a mixed fleet works and one
-worker's state never changes another's. The flag comes from the `/server_info`
-launch record, which has reported it all along, so no engine change is needed.
-A worker whose `/server_info` does not answer has no readable protocol and
-forwards over HTTP/1.1 for as long as it stays registered — a throughput cost,
-never a correctness one. Admin fan-out (`/flush_cache`) always uses the default
-client, because it addresses every worker at once rather than a selected one.
-
-Two things worth knowing when debugging. h2c is prior-knowledge only — there is
-no negotiation and no fallback — which is why the router requires the engine's
-own `enable_http2` report before using it. And a worker's protocol is fixed
-for as long as that worker stays registered: it is read once, at registration,
-and never re-read. Under the K8s backend a restarting engine flips its
-EndpointSlice to `ready=false`, which is a `Removed` → `Added` cycle and so a
-fresh reading; under `--worker-urls` the fan-out happens once at startup and
-nothing re-registers. So a worker registered over h2c that later stops serving
-it (a proxy interposed on its port, say) is not detected until it
-is re-registered; its circuit breaker will open in the meantime.
-
-## Native gRPC
-
-`--grpc-port` serves the engine's own gRPC API, `sglang.runtime.v1.SglangService`
-(`proto/sglang/runtime/v1/sglang.proto`), so a gRPC client can point at the
-router or at an engine. A call is routed exactly like its HTTP route and sent to
-the same RPC on the selected engine's native gRPC server, which engines run
-beside HTTP with their own `--grpc-port`; the router reads that port from each
-worker's `/server_info`. HTTP requests are unaffected and still reach engines
-over HTTP.
-
-| RPC | Routed like |
-|---|---|
-| `ChatComplete` | `/v1/chat/completions`: same rendering, `input_ids` forwarding, PD and DP-rank handling |
-| `Complete` | `/v1/completions` |
-| `Generate`, `TextGenerate` | `/generate` with `input_ids` or `text` |
-| `Embed`, `TextEmbed`, `OpenAIEmbed` | `/v1/embeddings` |
-| `Classify`, `OpenAIClassify` | `/v1/classify` |
-| `Rerank` | `/v1/rerank` |
-
-The OpenAI RPCs carry the HTTP route's JSON body, so they are prepared exactly as
-over HTTP. A typed RPC is prepared from its fields with the same rules, and the
-router's additions (rid, PD bootstrap, DP rank, sampling defaults) are set on the
-proto it forwards. A text prompt is tokenized once, as its HTTP route forwards
-`input_ids`: `TextGenerate` is sent as `Generate` with `return_text`, so the engine
-still supplies the text, and `TextEmbed` as `Embed`. They stay text RPCs under
-`--disable-input-ids-forwarding`, without a tokenizer, or, for `TextGenerate`, with
-`return_text_in_logprobs`. Other RPCs, such as
-`Tokenize`, `HealthCheck` and the admin RPCs, answer `UNIMPLEMENTED`. Request metadata and `trace_headers`
-act as HTTP headers for routing and forwarding, and a cancelled call aborts the
-engine request. Router errors arrive as a gRPC status carrying the HTTP path's
-message and an `x-router-error-code` trailer; an engine's status passes through
-unchanged. An engine without `--grpc-port` (including one launched with
-`--api-key`, which native gRPC rejects) cannot serve gRPC calls: the router warns
-when it registers, and calls routed to it fail with `UNAVAILABLE`.
-
-## Upgrading from `cache_aware_zmq`
-
-The `cache_aware_zmq` policy has been removed. Configurations using it should
-select `--policy cache_aware` and choose a native cache-prefix source: the
-Router-local radix tree (the default), or the external Indexer shown above.
-
-The legacy `--cache-threshold`, `--balance-abs-threshold`, and
-`--balance-rel-threshold` flags have also been removed. They do not have
-one-to-one replacements; remove them and review the current `sgl-router
---help` output when tuning Cache-Aware routing.
+- **No tokenizer**: `--no-tokenizer` for load-only policies (`power_of_two`,
+ `session_aware`); see [optional tokenizer](REFERENCE.md#optional-tokenizer-for-load-only-routing).
+- **Retries**: `--retry-max-attempts N` resends a request that failed before
+ any response to another worker; see [retries](REFERENCE.md#retries).
+- **DP-rank routing**: `--dp-aware` also picks the DP rank inside multi-rank
+ engines, and `--dp-rank-policy round_robin` cycles ranks that no routing key
+ or cached prefix decides; see [DP-rank routing](REFERENCE.md#dp-rank-routing).
+- **PD version groups**: `--pd-version-group-label ` pairs prefill and
+ decode only within the same engine version; see
+ [PD version groups](REFERENCE.md#pd-version-groups).
+- **Engine API key**: `--worker-api-key` for engines launched with `--api-key`;
+ see [engines with `--api-key`](REFERENCE.md#engines-with---api-key).
+- **Sampling contract**: `--override-sampling-params ''` pins sampling
+ parameters fleet-wide; `--sampling-param-conflict reject|allow` decides what
+ happens to requests that differ. See
+ [fleet-wide sampling contract](REFERENCE.md#fleet-wide-sampling-contract)
+ for parameter domains and null handling.
+- **Chat rendering**: use the same model files and chat-template settings on
+ router and workers, or set `--disable-input-ids-forwarding`.
+ `--tokenizer-backend fast` and `--tokenizer-l1-cache-mb N` speed up
+ tokenization.
+- **HTTP/2**: no configuration. h2c is used inbound when the client speaks it,
+ and outbound to workers that report `enable_http2`; see
+ [HTTP/2](REFERENCE.md#http2).
+- **Upgrading**: the `cache_aware_zmq` policy and the legacy cache and balance
+ threshold flags were removed; see
+ [upgrading](REFERENCE.md#upgrading-from-cache_aware_zmq).
## License
diff --git a/experimental/sgl-router/REFERENCE.md b/experimental/sgl-router/REFERENCE.md
new file mode 100644
index 0000000000..b4a5d50214
--- /dev/null
+++ b/experimental/sgl-router/REFERENCE.md
@@ -0,0 +1,640 @@
+# sgl-router reference
+
+Detailed behavior of the features summarized in [README.md](README.md). Routing
+design, policies and bucket configuration are in [ARCHITECTURE.md](ARCHITECTURE.md).
+
+- [Discovery and KV signals](#discovery-and-kv-signals)
+- [Bucket file membership](#bucket-file-membership)
+- [Routing and reliability](#routing-and-reliability)
+- [Chat rendering](#chat-rendering)
+- [Native `/generate`](#native-generate)
+- [OpenAI over /generate](#openai-over-generate)
+- [Embeddings](#embeddings), [Classify](#classify), [Completions](#completions), [Rerank](#rerank)
+- [Anthropic Messages](#anthropic-messages)
+- Per-model rendering: [DeepSeek V4 and V4.1](#deepseek-v4-and-v41), [GLM-5.3](#glm-53), [Qwen3.8](#qwen38), [Kimi-K3](#kimi-k3)
+- [HTTP/2](#http2)
+- [Native gRPC](#native-grpc)
+- [Upgrading from `cache_aware_zmq`](#upgrading-from-cache_aware_zmq)
+
+## Discovery and KV signals
+
+### PD version groups
+
+To run two engine versions side by side in PD mode (e.g. during a rollout),
+pass `--pd-version-group-label `. A prefill worker is then paired only with
+decode workers that have the same value for that EndpointSlice label (inherited
+from the Service, so use one Service per role and version), so KV never crosses
+versions. Unlabeled workers form their own group.
+
+```bash
+sgl-router --model-id qwen3 --service-discovery \
+ --prefill-selector app=engines-qwen3,role=prefill \
+ --decode-selector app=engines-qwen3,role=decode \
+ --pd-version-group-label sglang.ai/version-group
+```
+
+### External KV indexer
+
+```bash
+sgl-router \
+ --model-id qwen3 \
+ --tokenizer-path /models/qwen3/tokenizer.json \
+ --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
+ --policy cache_aware \
+ --cache-prefix-provider indexer \
+ --kv-indexer-endpoint http://10.0.0.10:50051 \
+ --kv-indexer-query-timeout-ms 100 \
+ --kv-indexer-query-max-inflight 32
+```
+
+The Indexer replaces the Router-local radix tree as the native Cache-Aware
+signal. Query timeouts and local concurrency are bounded by the two Indexer
+options, which default to 100 ms and 32 respectively.
+
+### LoRA cache identity
+
+LoRA routing requires workers that publish `BlockStored.lora_name`. Requests
+select the adapter through `model: "base:adapter"` on OpenAI routes, or
+`lora_path` on native routes. Salted generation requests also carry `cache_salt`.
+
+Worker block IDs are opaque and may differ across adapter loads or workers.
+The router derives adapter-aware lookup keys from event tokens and keeps a
+per-worker, per-DP-rank mapping for parent links and removals. Multiple block
+IDs can own the same lookup key; removing one leaves the other owners and
+storage tiers intact. The remote indexer bridge performs the same translation.
+Base-model events retain their existing published-key path.
+
+New replicas request `?include_lora=true` snapshots containing LoRA mappings
+(version 2). Legacy consumers receive a base-only version-1 view, so mixed
+snapshots do not prevent base-cache bootstrap during rolling upgrades. Base-only
+snapshots retain version 1 for every consumer. LoRA stores buffered during bootstrap retain their
+tokens; the original base-model buffering and replay behavior is unchanged.
+
+Reloading different adapter weights under the same public name can still
+credit stale engine blocks until their eviction. Names identify adapters;
+versioned names are needed to distinguish different weights.
+
+### KV-event gap replay
+
+ZMQ drops KV events at the publisher's high-water mark, and a lost removal
+leaves the router crediting a worker with blocks it no longer holds. Give each
+engine a replay socket and the router re-fetches the batches a sequence gap
+skipped:
+
+```bash
+python -m sglang.launch_server ... \
+ --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558"}'
+```
+
+Rank r publishes on 5557 + r and replays on 5558 + r, so with `dp_size > 1`
+place the replay base at least dp_size ports past the PUB base. The engine keeps
+the last `buffer_steps` batches (default 10000); older gaps stay unrepaired.
+`sgl_router_kv_event_replays_total{outcome}` counts the attempts.
+
+### Pending prefixes
+
+Until the engine's KV events arrive, requests that share a cold prefix see no
+cached owner and spread across workers, each prefilling the same prefix. This
+is common with parallel sampling, RL rollouts and agent fan-out.
+`--cache-pending-prefix-ttl-ms` credits a worker with a prompt's prefix for that
+long after routing it there, so the burst stays together. It is off by default
+and needs `--policy cache_aware` with the Router-local radix tree.
+`sgl_router_cache_pending_prefix_hits_total` counts lookups where a pending
+prefix matched deeper than any confirmed one; the policy's candidate and
+admission filters still decide whether that worker is picked.
+
+### Peer bootstrap (Kubernetes)
+
+A replica that starts mid-fleet subscribes to each worker's KV topic
+mid-stream, so everything already resident in the engines' caches is invisible
+to it and it routes cache-blind until traffic re-stores those blocks —
+degrading the engines' locality for the whole fleet, once per replica on every
+rolling update. With a peer selector set, a booting replica instead pulls a
+tree snapshot from a warm sibling over `/internal/kv_snapshot` and splices it
+under its own live delta stream, and `/readyz` stays 503 until that bootstrap
+settles. Off unless `--kv-peer-selector` is set; requires `--policy
+cache_aware` with the Router-local radix tree (not an external Indexer) and
+`--service-discovery`, which the CLI enforces.
+
+```bash
+sgl-router \
+ --model-id qwen3 --policy cache_aware \
+ --service-discovery --service-discovery-namespace prod \
+ --selector app=engines-qwen3 \
+ --kv-peer-selector kubernetes.io/service-name=sgl-router
+```
+
+- `--kv-peer-selector` is matched against **EndpointSlice labels** — the
+ router Service's labels plus `kubernetes.io/service-name`, never the pods'
+ labels, so `kubernetes.io/service-name=` is the unambiguous
+ choice. A pod-template label matches nothing and every replica boots cold.
+- `--kv-bootstrap-timeout-ms` (default 600000) bounds the whole bootstrap;
+ readiness waits on it, so startup/readiness probes must tolerate a replica
+ staying unready this long.
+- `--kv-bootstrap-fetch-timeout-cap-ms` (default 300000) caps one snapshot
+ fetch within that budget.
+- `--kv-bootstrap-seed-required` (off by default) keeps holding `/readyz` when
+ siblings were present but their tree could not be pulled, bounded at
+ max(3× the bootstrap timeout, 60s) — it delays a failed seed's replica (and
+ with it a rolling update) rather than shipping it cache-blind; it cannot
+ stall a rollout indefinitely.
+
+The deployment needs two things beyond the flags: the router's ServiceAccount
+must hold `get`/`list`/`watch` on `endpointslices` in the watched namespace
+(the worker-discovery Role already grants this), and the pod spec should wire
+`POD_NAME`, `POD_NAMESPACE` and `POD_IP` from the downward API so a replica
+can exclude itself from its own peer list. A rolling update should surge
+(`maxUnavailable: 0`) so new replicas always have warm siblings to copy from.
+`tests/e2e/k8s_integration/manifests/kv-bootstrap.yaml` is a worked example of
+all of it, and the `sgl_router_kv_bootstrap_*` series in
+[monitoring/README.md](monitoring/README.md) make the whole path observable.
+
+## Bucket file membership
+
+`worker_services` matches Kubernetes Services by `namespace/name`, using the
+`kubernetes.io/service-name` label on watched EndpointSlices. Replacement pods
+and new replicas join the same group automatically. The router's discovery
+selectors must include those EndpointSlices; this field does not expand the watch.
+A worker selected by several Services belongs to each of them.
+
+For static URL discovery, use `"worker_ids": ["http://worker:30000"]`: each ID
+is the configured worker URL. Kubernetes worker IDs are `namespace/pod-UID`
+(and change when a pod is replaced), so use `worker_services` for durable pools.
+Set only one membership field, or omit both for every engine of the group's role.
+Empty membership lists and blank worker IDs are rejected at startup.
+
+Admission fields left unset or set to `null` inherit CLI defaults. To apply a
+limit only to selected groups, omit that CLI default and set it on those groups.
+
+## Routing and reliability
+
+### Optional tokenizer for load-only routing
+
+`--no-tokenizer` skips tokenizer loading for load-only policies such as
+`power_of_two` and `session_aware`, on either routing path. Workers tokenize the
+original messages, and `/v1/tokenize` and `/v1/detokenize` are unavailable.
+Cache-aware routing, prefix-cache terms or filters, and buckets with token-length
+or context limits require a tokenizer. Reorg buckets that only select membership
+and load-based policies can use `--no-tokenizer`.
+
+### DP-rank routing
+
+An engine launched with `--dp-size` or `--attn-dp-size` runs several DP ranks,
+each with its own KV cache, behind one endpoint. With `--dp-aware`, the router
+also picks the rank inside the selected worker. It sends that rank as
+`X-Data-Parallel-Rank`, which the engine honors, and overwrites any value the
+client sent. The router picks the first of these that applies:
+
+1. A hash of the sticky routing key or session id, so a conversation keeps
+ its rank and router replicas agree.
+2. The rank with the deepest cached prefix in the local KV tree.
+3. The rank with the fewest requests this router has in flight on it, or with
+ `--dp-rank-policy round_robin`, the worker's ranks in turn.
+
+In PD mode, decode is ranked by step 3 only. The decode request names the
+prefill rank in `disagg_prefill_dp_rank`, and the bootstrap room satisfies
+`room % prefill_dp_size == prefill_rank`.
+
+### Retries
+
+`--retry-max-attempts N` (default 1, which disables retries; 3 is typical) lets
+a request that fails before any response reaches the client be sent again to a
+worker it has not tried yet. A failure is a transport error, an open circuit
+breaker, a 5xx, a 429, or a timeout: `--request-timeout-secs` bounds a whole
+non-streaming response, and a streaming one until its headers arrive. SGLang's
+chat endpoint sends those headers with the first token, so for streaming chat
+the same flag bounds time to first token, queueing included; raise it for
+long-context or deeply queued fleets. Once a streaming response's 2xx status
+has been sent, it is never retried, and neither is any other 2xx or 4xx. In PD mode the failed
+side is excluded, or both sides when the caller set `rid`, which an engine still
+running it would refuse; a new pair gets a new bootstrap room. Retries share the
+request's `--stale-request-timeout-secs` deadline, and none starts after it. When
+every eligible worker has failed, the client gets the last failure.
+`sgl_router_retries_total` counts the retried attempts.
+
+Each retry first waits out a jittered exponential backoff: a random delay in
+`[d/2, d]`, where `d` starts at `--retry-initial-backoff-ms` (default 50),
+doubles per retry, and is capped at `--retry-max-backoff-ms` (default 2000).
+Set the initial backoff to 0 to retry immediately. A backoff never waits past
+the request's stale deadline.
+
+### Engines with `--api-key`
+
+The router reads each worker's `/server_info` and `/model_info` to learn its
+model, KV-event publisher, HTTP/2 support and DP size. An engine launched with
+`--api-key` rejects those requests without the key, so pass the same key as
+`--worker-api-key`. Chat requests and `/flush_cache` forward the caller's
+`Authorization` header instead, so callers still need the engine key.
+
+### Fleet-wide sampling contract
+
+`--override-sampling-params` fixes the sampling configuration for every client
+of this router, independently of what the engine's own defaults happen to be
+(on native [`/generate`](#native-generate), in each prompt's `sampling_params`):
+
+```bash
+sgl-router \
+ --model-id qwen3 \
+ --tokenizer-path /models/qwen3/tokenizer.json \
+ --worker-urls http://10.0.0.1:30000 \
+ --override-sampling-params '{"temperature": 1, "top_p": 0.95, "n": 1}' \
+ --sampling-param-conflict reject
+```
+
+It takes one JSON object keyed by the request-body field names
+(`temperature`, `top_p`, `top_k`, `min_p`, `repetition_penalty`,
+`frequency_penalty`, `presence_penalty`, `n`). A configured value is injected
+whenever the request omits that field, so the engine's defaults cannot drift
+from what the operator declared. `temperature`, `top_p`, `top_k`, `min_p` and
+`repetition_penalty` are the five the engine resolves from the model's own
+`generation_config`, which is what makes them drift when a deployed image
+changes; the rest have fixed API defaults.
+
+An explicit `null` counts as omitting the field, not as a client-supplied
+value: the OpenAI API types these parameters as nullable with a documented
+default, so `null` asks for the default — and on a governed fleet the
+configured value is what the default is.
+
+For a request that does send a value, `--sampling-param-conflict` decides:
+`reject` (the default) 400s a differing value before admission, while `allow`
+forwards the client's value untouched — the router never silently rewrites what
+a client sent. A `reject` response carries
+`x-router-error-code: sampling_contract_violation` and is counted in
+`sgl_router_sampling_contract_rejections_total{param}`, so a rollout's blast
+radius is visible per parameter rather than folded into `bad_request`.
+
+`reject` also 400s a value it cannot read as a number, on a governed parameter
+only. The engine coerces more than JSON numbers — a bool and a numeric string
+both become numbers — by rules that are undocumented and need not match across
+a fleet, so a value the router cannot read is one it cannot prove conforms, and
+waving it through would make the pin bypassable. The common coercions are
+matched exactly (`false` is 0, `"0.5"` and `"1_0"` are 0.5 and 10), so this
+refuses only genuine garbage. `allow` is unaffected: it promises nothing, so
+such a value keeps flowing to the engine, which owns the request schema.
+
+A value may also be an inclusive band, `{"min": LO, "max": HI}`, for a
+parameter that stays tunable inside a range. A band names no value to inject,
+so it constrains only the requests that name the parameter; one that omits it
+gets the model's own `generation_config` default, which the router cannot see.
+Because a band can only ever reject, combining one with `allow` is a startup
+error.
+
+Values are range-checked at startup, so a misconfiguration fails the launch
+instead of 400ing every request at the engine. Repeating a key in the flag is
+also a startup error, rather than silently enforcing whichever copy came last.
+
+| parameter | accepted | notes |
+| --- | --- | --- |
+| `temperature` | `[0, 2]` | |
+| `top_p` | `(0, 1]` | |
+| `top_k` | `>= 1`, or exactly `-1` | `-1` is the engine's "whole vocabulary" spelling and its default. Being non-contiguous it cannot bound a band. Note `top_k: 1` is greedy decoding, not "disabled". |
+| `min_p` | `[0, 1]` | not an OpenAI parameter; the engine's domain |
+| `repetition_penalty` | `(0, 2]` | not an OpenAI parameter; the engine's domain |
+| `frequency_penalty` | `[-2, 2]` | |
+| `presence_penalty` | `[-2, 2]` | |
+| `n` | `[1, 128]` | |
+
+The OpenAI domains are deliberately narrower than what the engine itself
+accepts (`SamplingParams.verify` would take `temperature: 5`): these values are
+injected into request bodies, and a fleet contract outside the range every
+OpenAI client library validates against is far more likely a typo than an
+intent.
+
+Cost: a request that named every configured field forwards its original bytes
+untouched. One that omits a field has the scalars spliced directly into the
+body bytes — no parse, no re-serialize — which on a 16 MiB body is ~0.24 ms
+against ~5.7 ms for a `serde_json` round-trip. Only `input_ids` and PD
+bootstrap injection still parse and re-serialize, because those may have to
+overwrite a key the client sent.
+
+### Relationship to the engine's own `--preferred-sampling-params`
+
+The engine has an inject-when-absent flag of its own,
+`--preferred-sampling-params`, merged in
+`python/sglang/srt/managers/tokenizer_manager.py` as
+`{**preferred, **obj.sampling_params}`. On `/v1/chat/completions` it is
+currently a no-op: `ChatCompletionRequest.to_sampling_params`
+(`python/sglang/srt/entrypoints/openai/protocol.py`) resolves every sampling
+key eagerly through `generation_config` and then its own defaults, so the
+right-hand side of that merge is always fully populated and always wins.
+(`/v1/responses`, in the same file, already omits `None` entries for exactly
+this reason.) If that is fixed engine-side, `--preferred-sampling-params`
+covers the inject-when-absent half for a single engine.
+
+What stays the router's to own either way is the enforcement half: the
+`reject` immutability contract with its 400 before admission — costing no
+queue slot and no engine round-trip — the `sampling_contract_violation` code
+and per-parameter counter, bands, and one contract applied at a shared ingress
+across engines whose own flags the router operator may not control.
+
+## Chat rendering
+
+The router renders chat requests with dynamo-render (`dynamo-renderer`): the model's
+HF Jinja template from `tokenizer_config.json` or a sibling
+`chat_template.jinja`, or dynamo-render's built-in DeepSeek encoder (V4 family, V3.2)
+for template-less models. Cache-aware routing hashes the rendered tokens so its
+prefix queries match the blocks the engine caches. Models the engine encodes in
+code but dynamo-render cannot tokenize here (Inkling) route via raw prompt
+text, as does any model whose template fails to load or render.
+
+Some chats additionally forward the rendered tokens to the engine as
+`input_ids`, retaining the original messages, so the engine skips
+re-tokenizing. How many depends on the model's renderer. DeepSeek-V4's, V4.1's
+and Kimi-K3's native encoders and GLM-5.3's and Qwen3.8's templates are
+fixture-verified against SGLang's request normalization, so they forward every
+chat except multimodal ones and those with caller-provided `input_ids`. Renderers
+without that verification (HF Jinja templates)
+forward only plain text chat requests (string `content`, no tools, no template
+kwargs or reasoning controls or historical `reasoning_content`, no assistant
+continuation, no consecutive users or non-leading system turns) and warn
+`UNVERIFIED` at startup; every other request shape is rendered for routing
+only.
+
+Use matching model files on the router and workers, and set
+the same `--default-chat-template-kwargs`, `SGLANG_DEFAULT_THINKING`,
+`SGLANG_DSV4_REASONING_EFFORT`, and `SGLANG_DSV41_REASONING_EFFORT` on both.
+The router reads these render defaults from its own configuration and environment;
+it does not discover the workers' settings. Point `--tokenizer-path` at the workers'
+model snapshot so the V4 effort profile is read from the same
+`encoding/encoding_dsv4.py`.
+
+Set `--disable-input-ids-forwarding` for this router's model when worker-side
+rendering has not been verified to match. This disables router-generated IDs
+for every routing policy; cache-aware routing still renders and tokenizes
+locally, and the original messages reach the workers for engine processing.
+Caller-supplied `input_ids` remain caller-owned and pass through unchanged.
+
+Forwarding logs its assumptions at startup. In particular, disable it when the
+router's render defaults differ from the workers', for worker template overrides
+not reflected in the router's model files, for worker parser overrides such as
+`--tool-call-parser deepseekv32` that select a native encoder over a shipped
+template, or conversation templates with stop strings (the engine's `input_ids`
+path skips those template stops). Disabling forwarding preserves engine behavior
+but does not establish parity for local routing hashes.
+
+Also set `--disable-input-ids-forwarding` for array-only templates: Dynamo may wrap
+string content into arrays differently from the worker. The pinned Dynamo renderer does not expose
+its conversion flag, so the router cannot automatically block these templates.
+Detailed content-format parity coverage follows in #39133.
+
+The Dynamo crates are pinned exactly and `Cargo.lock` is committed; CI builds
+with `--locked`, so rendered bytes cannot change without a reviewed diff.
+
+Router tokenization sits on the TTFT path for every chat it renders. Two opt-in
+flags make it cheaper:
+
+- `--tokenizer-backend fast` encodes with fastokens (decoding stays on HF). It
+ needs a `tokenizer.json` and falls back to `hf` when fastokens cannot load it.
+- `--tokenizer-l1-cache-mb N` caches prefix tokenizations at special-token
+ boundaries, so a multi-turn chat encodes only the turns added since the
+ previous request. Boundaries are unconditional, non-normalized, non-stripping
+ special tokens with no overlapping added-token spellings. Unsafe candidates
+ are excluded; if none remain, encoding proceeds without the cache.
+
+On a ~69K-token DeepSeek-V4 chat, `hf` encodes in ~40 ms, `fast` in ~4 ms, and
+a new turn on a cached history in ~0.2 ms. The DeepSeek fixtures check every
+case under `hf`, `fast`, and `fast` with L1. Startup logs report the resolved
+backend and cache state. `/metrics` exposes only
+`sgl_router_tokenizer_l1_tokens_total{source="cached"|"encoded"}` to measure
+how much tokenization work the cache reuses.
+
+## Native `/generate`
+
+`/generate` (`POST` or `PUT`) has the engine's interface: the same
+`GenerateReqInput` body and the same response, buffered or SSE. The body names no
+model, so requests go to the one this router serves. Worker selection
+(`--chat-routing`, `--policy`), PD dispatch, and abort-on-disconnect are shared
+with chat completions.
+
+With a tokenizer loaded, the router tokenizes `text` (a string or a list) with
+the special tokens SGLang adds (BOS per `add_bos_token` for Llama-, Gemma- and
+Cohere-class tokenizers, otherwise the `tokenizer.json` post-processor's), and
+forwards it as `input_ids`. The engine skips tokenizing and routing sees its exact
+tokens. Multimodal requests keep `text`, since the engine expands placeholders
+from it, and `--disable-input-ids-forwarding` keeps it for every request. So does
+a model whose `tokenizer.json` normalizer transformers replaces on load (legacy
+SentencePiece Llama files, bge-m3), or whose `tokenizer_config.json` the router
+cannot read, as the router cannot reproduce its tokens. A batch goes to one
+worker: load counts every prompt and each of its `n` samples, while bucket and
+context limits bound the longest prompt plus its own `max_new_tokens`.
+
+Otherwise the body passes through, plus PD bootstrap fields, a minted `rid` for
+a single prompt that has none, and `--override-sampling-params` defaults. Under
+`--dp-aware` each worker's body also carries the chosen `routed_dp_rank`; a PD
+batch or `n > 1` request leaves the prefill rank to the engine, which gives item
+`i` the bootstrap room `room + i`.
+
+## OpenAI over /generate
+
+`/v1/completions` and `/v1/chat/completions` behave as the engine's own routes,
+but the router runs SGLang's OpenAI layer itself (`sglang-processor`'s `openai` module): it lowers
+the request to the `GenerateReqInput` SGLang would build, sends it to
+`/generate` like a native request, and builds the OpenAI response, buffered or
+SSE, from the engine's output. The engine's `/server_info` supplies the server
+args that layer reads (`--enable-cache-report`,
+`--stream-response-default-include-usage`, `--incremental-streaming-output`, the
+custom-labels header). Requests it cannot reproduce exactly go to the engine's
+own route as sent: workers that disagree on those args or never
+reported them, `--completion-template`, `return_hidden_states` and other
+`sglext` outputs, `echo` or `logprobs` without a router tokenizer, and bodies
+SGLang would reject. Chat also needs a renderer verified against SGLang (the
+`AllText` scope: DeepSeek-V4, V4.1, GLM-5.3, Kimi-K3 and Qwen3.8 today), router `--default-chat-template-kwargs`
+equal to the workers', a reasoning parser the processor reproduces
+(`deepseek-v4`, `deepseek-v41`, `glm45`, `kimi_k3`, `qwen3`) or none, and no media yet. Tools are served with `tool_choice`
+`auto` or `none`, non-strict, with standard JSON-schema types, and a tool
+parser the processor reproduces (`deepseekv4`, `deepseekv41`, `kimi_k3`, `qwen3_coder`); `required`, named and strict
+tools take a constraint from the engine's xgrammar, so they go to the engine
+(`kimi_k3`'s `auto` tag is reproduced).
+With `--tool-call-parser glm47`, a chat with tools also goes to the engine: its
+grammar needs `ebnf_full_assistant`, which `/generate` does not accept.
+The router reads the model's `config.json` and `generation_config.json` beside
+`--tokenizer-path` (or from the HF repo), as the engine reads them, so a local
+path should point into the model's directory. Engine env vars that change the
+layer (`SGLANG_TOOL_STRICT_LEVEL`, `SGLANG_FORWARD_UNKNOWN_TOOLS`) are taken as
+unset. `sgl_router_openai_route_total` counts each outcome.
+
+## Embeddings
+
+`/v1/embeddings` has the engine's interface: the same OpenAI `EmbeddingRequest`
+body and response. As for chat completions, `model` must name the served model.
+A PD fleet answers 400, since prefill and decode engines serve no embeddings.
+
+Text `input` (a string or a list) is tokenized and forwarded as token IDs, as for
+[`/generate`](#native-generate), plus the EOS SGLang appends for EmbeddingGemma.
+Blank prompts and multimodal items stay as sent, for the engine to reject or
+render. A list is a batch for one worker, routed on load. A single prompt with no
+`rid` gets a minted one. Under `--dp-aware` the engine picks the rank, since its
+embeddings endpoint reads none.
+
+## Classify
+
+`/v1/classify` has the engine's interface: the same `ClassifyRequest` body and
+response. It is served as embeddings are, except that a batch of text stays text,
+since the engine takes token IDs for only one prompt.
+
+## Completions
+
+`/v1/completions` has the engine's interface and is prepared like chat (sampling
+contract, PD bootstrap, DP rank), with the prompt forwarded as sent: tokens the
+router computes for routing do not replace it.
+
+## Rerank
+
+`/v1/rerank` has the engine's interface: the same `V1RerankReqInput` body and
+response, sent as `POST` or `PUT` to the model this router serves. The body is
+forwarded as sent, since the engine renders and tokenizes each query-document
+pair. Routing is on load, with each pair's size estimated from its text. As for
+embeddings, a PD fleet answers 400 and `--dp-aware` pins no rank.
+
+## Anthropic Messages
+
+`/v1/messages` converts each request to a chat completion at ingress, so it is
+routed exactly like `/v1/chat/completions`, and converts the reply (or stream)
+back to the Messages format. Consecutive turns of the same role are merged, as
+the Messages API does, and a final assistant turn is sent as a prefill
+(`continue_final_message`). Besides the Messages fields, only
+`chat_template_kwargs` and `continue_final_message` reach the chat request.
+Every error uses the Anthropic error envelope; 5xx messages are replaced with a
+generic one, and a failed stream still closes its open block and ends with
+`message_stop`.
+
+`/v1/messages/count_tokens` is answered by the router from the model's chat
+template, without calling an engine. Image tokens are not counted, and a model
+without a chat template answers 500 rather than a raw-text estimate.
+
+`thinking` sets the `thinking` / `enable_thinking` chat template kwargs. The
+engine's own `/v1/messages` also adapts it per reasoning parser (for example
+`reasoning_effort` for Hunyuan), which the router cannot see; on such models
+use `output_config.effort` or `chat_template_kwargs` directly.
+
+## DeepSeek V4 and V4.1
+
+Native V4 and V4.1 rendering comes from `rust/sglang-processor`, shared with SGLang's Rust
+server, and follows SGLang's serving path (`serving_chat.py`), not Dynamo's
+OpenAI defaults: all declared tools are rendered with SGLang's schema
+defaults, reasoning effort comes from `reasoning` / `reasoning_effort`, and the
+official/preview effort profile is detected from the checkpoint's
+`encoding/encoding_dsv4.py` or overridden by `dsv4_reasoning_effort_profile` in
+`config.json`, as in SGLang. Reference prompts live in `tests/fixtures/deepseek/`
+and are regenerated by `tests/scripts/generate_deepseek_parity.py`.
+
+V4.1 Flash uses Dynamo's V4.1 encoder with SGLang's numeric reasoning budgets
+and tool payloads. Developer messages and media are left to the worker, since
+the encoder renders them differently.
+
+## GLM-5.3
+
+GLM-5.3 and GLM-5.3-Flash render with the checkpoint's `chat_template.jinja` through
+`rust/sglang-processor`, which builds the template context SGLang's Jinja path builds,
+so every text chat forwards `input_ids`. Media, a `chat_template` kwarg, `-0` in tools
+or tool arguments, and U+001C..U+001F in assistant history render on the worker. With
+the cookbook's parsers, chats without tools are served through `/generate`; chats with
+tools keep the engine's route (see above).
+
+## Qwen3.8
+
+Qwen3.8 (Max, 27B and Flash-Next) renders with the checkpoint's `chat_template.jinja`
+through `rust/sglang-processor`, which picks it by its template and builds the
+template context SGLang's Jinja path builds. Every text chat forwards `input_ids`.
+Media, a `chat_template` key in `chat_template_kwargs`, and text or tool numbers
+Python's `trim` or `json.dumps` treat unlike Dynamo's (such as `1e-06`) render on the
+worker. With the cookbook's `qwen3` and `qwen3_coder` parsers, text chats are served
+through `/generate`; thinking follows the template's `enable_thinking` (on by default).
+
+## Kimi-K3
+
+Kimi-K3 renders through `rust/sglang-processor` (dynamo-render's XTML formatter
+with SGLang's request semantics) and is tokenized with the checkpoint's chunked
+tiktoken encoding; shapes the formatter renders differently go to the worker.
+`--tokenizer-path` accepts a
+local `tiktoken.model` or an HF repo id, whose `tiktoken.model`, `config.json`
+and `tokenizer_config.json` are downloaded when it has no `tokenizer.json`.
+With the cookbook's `kimi_k3` parsers, text chats are served through `/generate`.
+
+## HTTP/2
+
+There is nothing to configure. The router negotiates per connection inbound and
+resolves the protocol per worker outbound; every combination below is reached
+automatically, and HTTP/1.1 remains a supported peer on both edges.
+
+**Inbound.** The listener accepts cleartext HTTP/2 (h2c, prior knowledge) and
+HTTP/1.1 on the same `--port`, chosen per connection. A mesh sidecar or
+load balancer that prefers h2c multiplexes over one connection instead of
+opening one per request; an HTTP/1.1 client is unaffected.
+
+**Outbound.** At registration the router reads each worker's `/server_info` and
+forwards over cleartext h2c only when that worker reports `--enable-http2` (the
+engine's Granian server, which serves h2c and HTTP/1.1 together) **and** the
+worker URL is cleartext. Everything else uses the default client, which speaks
+HTTP/1.1 in cleartext and negotiates ALPN `h2, http/1.1` over TLS:
+
+| `/server_info` | worker URL | router forwards over |
+|---|---|---|
+| `enable_http2: true` | `http://` | cleartext h2c |
+| `enable_http2: true` | `https://` | HTTP/2 over TLS, by ALPN |
+| `enable_http2: false`, or absent | any | HTTP/1.1 (cleartext) or ALPN (TLS) |
+
+The choice is per worker, not fleet-wide, so a mixed fleet works and one
+worker's state never changes another's. The flag comes from the `/server_info`
+launch record, which has reported it all along, so no engine change is needed.
+A worker whose `/server_info` does not answer has no readable protocol and
+forwards over HTTP/1.1 for as long as it stays registered — a throughput cost,
+never a correctness one. Admin fan-out (`/flush_cache`) always uses the default
+client, because it addresses every worker at once rather than a selected one.
+
+Two things worth knowing when debugging. h2c is prior-knowledge only — there is
+no negotiation and no fallback — which is why the router requires the engine's
+own `enable_http2` report before using it. And a worker's protocol is fixed
+for as long as that worker stays registered: it is read once, at registration,
+and never re-read. Under the K8s backend a restarting engine flips its
+EndpointSlice to `ready=false`, which is a `Removed` → `Added` cycle and so a
+fresh reading; under `--worker-urls` the fan-out happens once at startup and
+nothing re-registers. So a worker registered over h2c that later stops serving
+it (a proxy interposed on its port, say) is not detected until it
+is re-registered; its circuit breaker will open in the meantime.
+
+## Native gRPC
+
+`--grpc-port` serves the engine's own gRPC API, `sglang.runtime.v1.SglangService`
+(`proto/sglang/runtime/v1/sglang.proto`), so a gRPC client can point at the
+router or at an engine. A call is routed exactly like its HTTP route and sent to
+the same RPC on the selected engine's native gRPC server, which engines run
+beside HTTP with their own `--grpc-port`; the router reads that port from each
+worker's `/server_info`. HTTP requests are unaffected and still reach engines
+over HTTP.
+
+| RPC | Routed like |
+|---|---|
+| `ChatComplete` | `/v1/chat/completions`: same rendering, `input_ids` forwarding, PD and DP-rank handling |
+| `Complete` | `/v1/completions` |
+| `Generate`, `TextGenerate` | `/generate` with `input_ids` or `text` |
+| `Embed`, `TextEmbed`, `OpenAIEmbed` | `/v1/embeddings` |
+| `Classify`, `OpenAIClassify` | `/v1/classify` |
+| `Rerank` | `/v1/rerank` |
+
+The OpenAI RPCs carry the HTTP route's JSON body, so they are prepared exactly as
+over HTTP. A typed RPC is prepared from its fields with the same rules, and the
+router's additions (rid, PD bootstrap, DP rank, sampling defaults) are set on the
+proto it forwards. A text prompt is tokenized once, as its HTTP route forwards
+`input_ids`: `TextGenerate` is sent as `Generate` with `return_text`, so the engine
+still supplies the text, and `TextEmbed` as `Embed`. They stay text RPCs under
+`--disable-input-ids-forwarding`, without a tokenizer, or, for `TextGenerate`, with
+`return_text_in_logprobs`. Other RPCs, such as
+`Tokenize`, `HealthCheck` and the admin RPCs, answer `UNIMPLEMENTED`. Request metadata and `trace_headers`
+act as HTTP headers for routing and forwarding, and a cancelled call aborts the
+engine request. Router errors arrive as a gRPC status carrying the HTTP path's
+message and an `x-router-error-code` trailer; an engine's status passes through
+unchanged. An engine without `--grpc-port` (including one launched with
+`--api-key`, which native gRPC rejects) cannot serve gRPC calls: the router warns
+when it registers, and calls routed to it fail with `UNAVAILABLE`.
+
+## Upgrading from `cache_aware_zmq`
+
+The `cache_aware_zmq` policy has been removed. Configurations using it should
+select `--policy cache_aware` and choose a native cache-prefix source: the
+Router-local radix tree (the default), or the external Indexer shown above.
+
+The legacy `--cache-threshold`, `--balance-abs-threshold`, and
+`--balance-rel-threshold` flags have also been removed. They do not have
+one-to-one replacements; remove them and review the current `sgl-router
+--help` output when tuning Cache-Aware routing.
diff --git a/experimental/sgl-router/assets/architecture-deployment.png b/experimental/sgl-router/assets/architecture-deployment.png
new file mode 100644
index 0000000000..d5f7d0b01b
Binary files /dev/null and b/experimental/sgl-router/assets/architecture-deployment.png differ
diff --git a/experimental/sgl-router/src/config/cli.rs b/experimental/sgl-router/src/config/cli.rs
index 43ee432b38..0bcd8cf92f 100644
--- a/experimental/sgl-router/src/config/cli.rs
+++ b/experimental/sgl-router/src/config/cli.rs
@@ -100,7 +100,7 @@ pub struct ModelArgs {
/// Accepts temperature, top_p, top_k, min_p, repetition_penalty,
/// frequency_penalty, presence_penalty, and n. Numeric values fill absent
/// fields; {"min": LO, "max": HI} bands only constrain supplied values
- /// and require reject mode. See README for domains and null handling.
+ /// and require reject mode. See REFERENCE.md for domains and null handling.
#[arg(long, value_name = "JSON")]
pub override_sampling_params: Option,
diff --git a/experimental/sgl-router/src/policies_reorg/mod.rs b/experimental/sgl-router/src/policies_reorg/mod.rs
index b0873c14d1..7cb23a88d4 100644
--- a/experimental/sgl-router/src/policies_reorg/mod.rs
+++ b/experimental/sgl-router/src/policies_reorg/mod.rs
@@ -1,7 +1,7 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 The SGLang Authors
// SPDX-License-Identifier: Apache-2.0
-//! Side-by-side implementation of POLICY_DESIGN.md. Chat routing can opt into
+//! Routing policies described in ARCHITECTURE.md. Chat routing can opt into
//! this interface through AppContext; `policies` remains the default.
pub mod admission;