diff --git a/experimental/sgl-router/ARCHITECTURE.md b/experimental/sgl-router/ARCHITECTURE.md new file mode 100644 index 0000000000..b50a20af91 --- /dev/null +++ b/experimental/sgl-router/ARCHITECTURE.md @@ -0,0 +1,236 @@ +# SGLang router architecture + +This document describes `--chat-routing reorg`: the deployment, request flow, +and routing policies. The default legacy path is outside its scope. + +- [1. Overview](#1-overview) +- [2. Configuration examples](#2-configuration-examples) + +## 1. Overview + +### 1.1 Deployment Architecture + +A deployment consists of a **router** and **buckets of SGLang engines**. A bucket +is a logical group defined in the router configuration. It contains either a +**unified engine group**, where each engine handles both prefill and decode, +or separate **prefill and decode (P/D) groups**. + +The example below shows short-context and long-context buckets, each with prefill +and decode engines. + +![Router deployment with short-context and long-context P/D engine buckets](./assets/architecture-deployment.png) + +Each engine publishes load reports and KV events, which the router uses for +routing. An optional [external KV indexer](sgl-kv-indexer/README.md) can supply +prefix matches. The router also maintains session bindings and in-flight request +counts. + +To serve a request, the router chooses a bucket, selects engines within it, and +forwards the request. In P/D, it sends the request to both the prefill and decode +engines; the prefill engine transfers KV directly to the decode engine. + +### 1.2 Request routing lifecycle +Routing a request typically goes through **preprocessing** (rendering and +tokenizing), **picking a bucket**, **scoring candidate engines** (load, prefix +matching, or session affinity), **choosing engine(s)**, **forwarding** the +request, and **handling output tokens** (e.g. tool parsing). Along the way it +also applies **admission control**, **failure handling**, and **retries**. + +### 1.3 Example: load-aware routing + +The router tries each **bucket** that fits the request's length, ordered by SLO +preference, capacity, then `rank`; a request that fits no bucket is rejected +with **400** before any engine is considered. In each bucket it drops the +engines that fail **admission**, then picks the **less loaded of two engines +sampled** from those that remain; when no engine admits the request it moves to +the next bucket, and returns **503** when no bucket has one. If **dispatch** +fails before any response and **retries** remain, it excludes that engine and +restarts from the first bucket. + +**Load** is a vector of engine metrics compared field by field, and the first +difference decides. For **unified and prefill** engines the order is estimated +prefill queue time, waiting uncached tokens, waiting requests, running requests, +then in-flight requests; **decode** engines rank by waiting and running requests, +then KV usage. + +```mermaid +%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#eaf3fb", "primaryTextColor": "#172b4d", "primaryBorderColor": "#345779", "lineColor": "#345779", "edgeLabelBackground": "#ffffff"}, "flowchart": {"curve": "stepAfter", "rankSpacing": 30, "nodeSpacing": 30}}}%% +flowchart TD + R([Request]) --> L{"0. Fits any bucket's
length limits?"} + L -->|no| Y(["400 to client"]) + L -->|yes| B["1. Take next compatible bucket"] + B --> A{"2. Any engine can admit
the request?"} + A -->|yes| S["3. Sample two admitted engines
Pick the less loaded one"] + S --> D["4. Dispatch to engine"] + A -->|no| M{"More buckets?"} + M -->|no| X(["503 to client"]) + D -->|success, or
out of retries| E([Response to client]) + D -->|failed before response,
retries left| T["Exclude failed engine"] + M -.->|yes: next bucket| B + T -.->|restart at first bucket| B + classDef next fill:#fff5e6,stroke:#ac762c,color:#654516; + classDef done fill:#e8f5ef,stroke:#46826a,color:#234d3c; + class T,X,Y next; + class E done; +``` + +### 1.4 Example: cache-aware and session-aware routing + +These policies replace steps **2–3** above. `cache_aware` serves plain and +prefill groups only; decode groups use load-aware (`power_of_two`) unless they set +`session_aware`. Both policies split the group's engines in two: + +- **Preferred engine (A):** for `cache_aware`, the best-ranked prefix owner that + passes admission; for `session_aware`, the engine bound to the session. +- **Normal engines:** the rest of the engine group. When needed, the router picks + one of them (**B**) with the load-aware `power_of_two` policy. + +A and B are checked against the same group admission limits (for example +`max_kv_usage` or `max_waiting_requests`; see [configuration examples](#2-configuration-examples)). + +- **`prefer` mode (default):** use A if it passes admission. +- **`balanced` mode:** also pick a B engine and compare the loads of A and B. + - Switch to B only when it is clearly less loaded: + `load(A) > load_factor × load(B)` **and** `load(A) − load(B) > load_gap` + (defaults: factor 2; gap 1024 tokens, or 4 requests). Otherwise, including + when either load is unknown, keep A. + - `balanced_by` sets the load metric: + - **`prefill_tokens` (default):** the engine's reported waiting uncached tokens, + plus the tokens this request would prefill on it. For `cache_aware` that + subtracts the engine's cached prefix, so A's cache hit counts in its favor. + - **`running_requests`:** the engine's reported running requests. +- **No A, or A fails admission (either mode):** fall back to B. + +## 2. Configuration examples + +**CLI flags** set the defaults for every engine group. A **bucket file** +(`--bucket-config`) defines the buckets, and each group in it can override the +policy, admission, and affinity settings for itself. Without a bucket file, +there are no length tiers: every request can go to any engine. + +Settings fall into these categories: + +| Category | What it controls | CLI flags | Bucket-file group key | +| --- | --- | --- | --- | +| **Engines** | Plain or P/D, and where engines come from | `--worker-urls`, or `--service-discovery` with `--selector` (plain) or `--prefill-selector` + `--decode-selector` (P/D) | `worker_services`, `worker_ids` | +| **Policy** | How the router scores engines within a group | `--policy` | `policy` | +| **Affinity** | When the preferred engine (prefix owner or session binding) wins over a normal engine | `--affinity-*`, `--cache-affinity-*`, `--session-id-header` | `affinity` | +| **Admission** | Limits that make an engine reject a request | `--max-*` | `admission` | +| **Buckets** | Which buckets a request fits and the order they are tried | `--bucket-config` | `max_input_tokens`, `max_context_tokens`, `rank`, ... | + +How policies apply to each group: + +- **Plain and prefill groups** use `--policy` (default `power_of_two`). +- **Decode groups** always default to `power_of_two`; under reorg, + `--decode-policy` must stay `power_of_two`. +- **A bucket-file group** can set `"policy"` to `power_of_two` or to the + `--policy` value, nothing else. `cache_aware` cannot be used on decode, so + only `session_aware` can replace `power_of_two` there. + +### Example A: plain engines, cache-aware + +```bash +sgl-router --model-id qwen3 --chat-routing reorg \ + --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \ + --policy cache_aware \ + --affinity-mode prefer --cache-affinity-min-match-ratio 0.2 \ + --max-waiting-requests 16 --max-in-flight 56 +``` + +- **Engines:** two plain engines; each handles both prefill and decode. +- **Policy:** every engine is scored with `cache_aware`. +- **Affinity:** + - An engine owns the prefix if it holds at least 1024 matched tokens (the + default) **and** 20% of the prompt. + - `prefer` sends the request to the best owner that passes admission. +- **Admission:** an engine is rejected when it reports 16 or more waiting + requests, or has 56 or more requests in flight from this router. + +### Example B: P/D engines, session-aware + +```bash +sgl-router --model-id qwen3 --chat-routing reorg \ + --service-discovery --service-discovery-namespace inference \ + --prefill-selector app=qwen3,role=prefill \ + --decode-selector app=qwen3,role=decode \ + --policy session_aware --session-id-header x-session-id \ + --affinity-mode balanced --affinity-balanced-by running-requests \ + --max-kv-usage 0.9 +``` + +- **Engines:** Kubernetes discovery finds prefill and decode engines by label. +- **Policy:** + - Prefill engines use `session_aware`: requests with the same + `x-session-id` stick to the prefill engine they are bound to. + - Decode engines use `power_of_two`. To make decode session-aware too, use a + bucket file whose decode group sets `"policy": "session_aware"`. +- **Affinity:** in balanced mode the router samples an alternative engine, and + the session moves to it when the bound engine has more than 2x the + alternative's running requests **and** more than 4 more than it (the + defaults). The session also moves when the bound engine fails admission. +- **Admission:** prefill and decode engines are rejected at 90% KV usage. + +### Example C: P/D buckets from a file + +```bash +sgl-router --model-id qwen3 --chat-routing reorg \ + --service-discovery --service-discovery-namespace inference \ + --prefill-selector app=qwen3,role=prefill \ + --decode-selector app=qwen3,role=decode \ + --policy cache_aware \ + --bucket-config buckets.json +``` + +`buckets.json` defines a short tier with a TTFT estimate and a long tier: + +```json +{ + "ttft_slo": "slo_first", + "buckets": [ + { + "id": "short", + "max_input_tokens": 4096, + "max_context_tokens": 8192, + "ttft_ms": 300, + "prefill": { + "worker_services": ["inference/prefill-short"], + "admission": {"max_pending_prefill_tokens": 32768}, + "affinity": {"mode": "balanced", "load_factor": 1.5} + }, + "decode": { + "worker_services": ["inference/decode-short"], + "admission": {"max_kv_usage": 0.9} + } + }, + { + "id": "long", + "rank": 1, + "max_context_tokens": 131072, + "prefill": { + "worker_services": ["inference/prefill-long"], + "policy": "power_of_two" + }, + "decode": {"worker_services": ["inference/decode-long"]} + } + ] +} +``` + +- **Engines:** each group takes the discovered engines of its role that belong + to the listed Kubernetes Services. +- **Buckets:** + - Requests that fit within 4k input and 8k context fit both buckets; `short` + is tried first because it is smaller. Longer requests go only to `long`. + - If `short` rejects a request, the router tries `long`. +- **Policy:** + - `short` prefill uses `cache_aware` from `--policy`. + - `long` prefill overrides it with `power_of_two`. + - Both decode groups use `power_of_two`. +- **Affinity:** `short` prefill uses balanced mode with a load factor of 1.5. + `power_of_two` groups take no `affinity` block; setting one is rejected at + startup. +- **Admission:** + - `short` prefill engines reject requests while 32k or more uncached tokens + are waiting. + - `short` decode engines reject requests at 90% KV usage. + - `long` engines set no limits. diff --git a/experimental/sgl-router/POLICY_DESIGN.md b/experimental/sgl-router/POLICY_DESIGN.md deleted file mode 100644 index 6c93e00605..0000000000 --- a/experimental/sgl-router/POLICY_DESIGN.md +++ /dev/null @@ -1,655 +0,0 @@ -# Engine selection: target design - -This document describes the target architecture for engine selection. It defines -responsibilities and behavior; the interface and configuration examples are -sketches, not a specification of the current API or CLI. Implementation status is -listed at the end. - -## Principles - -1. **Filter buckets by token length, then order preferences.** `BucketResolver` - returns all compatible buckets, applying optional SLO preferences before - capacity/rank/ID ordering, without inspecting workers or policies. -2. **The bucket owns plain versus PD engine selection.** `Bucket::pick_engines` - calls its one plain group or both prefill and decode groups, returning a - complete selection. Both PD engines come from that same bucket. -3. **An engine group owns membership and its policy.** `EngineGroup::pick` - filters live workers by model, health, stage, and membership, invokes the - policy, and validates that its result belongs to the exact candidate set. -4. **A policy owns selection, fallback, admission, and its state dependencies.** - Construction injects the load, KV, or affinity handles it needs. Request-time - arguments contain facts and candidates. A policy cannot choose another bucket - or cross a PD role boundary. -5. **The handler owns bucket fallback and dispatch.** It calls `pick_engines` - on each bucket and dispatches only after a complete selection succeeds. - Missing candidates or admission rejection advance to the next bucket, where - all required engines are selected again. Invalid signals or policy results stop routing. - -An engine is represented by `Worker`. Registry role labels remain authoritative -when filtering candidates. Groups reference worker IDs rather than owning live -workers. Group policy instances are reused across requests. - -### Data ownership - -| Type | Owns | -| --- | --- | -| `BucketResolver` | A model's bucket collection, length filtering, and ordering | -| `Bucket` | ID, length constraints, rank, groups, and complete plain/PD engine selection | -| `EngineGroup` | Engine membership, attached policy, and engine selection | -| `WorkerRegistry` | Live workers, model membership, health, and role | -| Request handler | Request preparation, ordered bucket attempts, HTTP errors, and dispatch | - -Omitting both `worker_ids` and `worker_services` includes every healthy engine -serving the requested model and role. Explicit empty membership lists are rejected -by the config factory. `EngineGroup::new(policy)` creates a catch-all group. -`BucketGroups::Pd` requires both groups, making partial or mixed plain/PD bucket -configurations unrepresentable. - -## 1. Code organization - -```text -src/ - buckets.rs Bucket resolution and engine-group selection - policies/ - mod.rs Policy contract and construction - admission.rs Per-engine acceptance checks - cache_aware.rs Cache selection and local/remote adapter - session_aware.rs Session selection and fallback - sticky.rs Routing-key selection and fallback - least_load.rs Least-load selection - power_of_two.rs Pair sampling and stage-aware comparison - random.rs Random selection - round_robin.rs Rotation - state/ - mod.rs Shared exports - kv_events/ Local index, subscriptions, hashing, wire format - load_monitor/ Engine reports and router-local request accounting - affinity_store.rs Assignments, expiry, and atomic updates - server/ - app_context.rs Shared service lifecycle and policy wiring - routes/chat.rs Request preparation, PD coordination, dispatch -``` - -During migration, `buckets_reorg.rs` and `policies_reorg/` implement this design -beside the live `policies/` path. The layout above is the target after switchover. -Shared state already lives directly under `src/state/`. - -Dependencies flow from engine groups to policies, and from policies and admission -to shared state. State does not depend on bucket ordering or concrete policy -strategies. Small shared helpers are sufficient; no generic score-composition, -tier executor, or separate selection framework is required. - -## 2. Responsibilities and request flow - -`BucketResolver::resolve(input_tokens, expected_peak_tokens, ttft_ms, tokens_per_second)` returns an ordered -list of compatible bucket references (possibly empty), or an invalid-signal error. -It does not receive a stage or a load view, resolve live engines, or invoke policies. -The handler iterates this list until a bucket supplies the complete engine selection. - -`Bucket::pick_engines(workers, request)` accepts a `BucketRequest` -of prepared routing facts, invokes the required groups, and returns `BucketPick` -(one plain pick or a complete P/D pair). Failures retain their stage. This API -has no HTTP headers, `AppContext`, or forwarding dependency. - -`EngineGroup::pick(workers, request)` resolves healthy workers for the request's -model and stage, intersects them with its membership, sorts them by stable ID, -and invokes its attached policy. It rejects foreign results, including a newly -allocated worker with the same ID as a candidate. - -```text -chat_completions (reorg configured): prepare tokens and expected peak - | - v -BucketResolver::resolve: ordered length-compatible buckets - | - v -For each bucket: bucket.pick_engines(...) - | - +-- BucketGroups::Plain - | plain.pick() -> one plain engine - | - +-- BucketGroups::Pd - prefill.pick() -> P engine - decode.pick() -> D engine from the same bucket - | - +-- empty group / admission rejection -> try next bucket (repeat all picks) - +-- invalid signal / configuration / foreign pick -> return error - | - v -Complete selection -> forward_chat_request: acquire guards, attach PD bootstrap, forward response -``` - -The handler extracts token facts and header keys once into `BucketRequest`. -The bucket creates a stage-specific `PickRequest` for each group call, supplying -its own ID and the role associated with that group. Input length, expected peak, -token IDs, and session/routing keys pass through. Policies obtain observations -from their own shared-state handles; buckets and handlers do not provide load, -KV, or affinity services on each call. PD uses separate group policies, but never -independently resolves a decode bucket. Decode selection -failure discards that tentative prefill choice and advances to the next bucket -on missing candidates or admission rejection. No forwarding guards are acquired -and no prefill request is sent until both picks in one bucket succeed. -PD compatibility constraints beyond model and role remain follow-up work. - -There is one endpoint: `POST /v1/chat/completions`. `AppContext::chat_routing` -chooses its implementation: - -- `ChatRouting::Legacy` (default) uses the existing policies and bucket selector. -- `ChatRouting::Reorg(HashMap)` uses the new bucket and - policy interfaces, with explicit model-specific resolvers. - -`--chat-routing reorg` constructs default plain/PD buckets using the existing -`--policy` and tuning flags. Programmatic callers can still install explicit -resolvers. A missing model returns 404 without falling back to legacy routing. -The default serving path remains legacy. - -Both implementations reuse request preparation (including sampling validation -and tokenization), forwarding, streaming, middleware, and the 32 MiB body limit. -The reorg implementation requests tokenization for length matching, retaining -the existing body-size estimate when tokenization is unavailable. - -## 3. Bucket resolution - -1. Validate that a known expected peak is at least the input length. -2. Keep buckets whose inclusive input-token range contains the input length. -3. Check the bucket context capacity against input plus requested output when - known, or against input length when the output budget is unknown. -4. Apply enabled SLO preferences to complete buckets. Count unmet preferences - equally, then sort by ascending input capacity (the lesser of the input upper - bound and context capacity), rank, and ID. Unbounded capacities sort last. - Return the entire ordered list, retaining nonpreferred buckets for fallback. -5. The handler calls each bucket's `pick_engines` until one supplies its complete - selection. A failed PD attempt never contributes an engine to a later pair. - -The handler checks addition overflow when computing the expected peak. -An empty bucket list becomes a 400 `NoMatchingBucket` response. After exhausting -the list, accumulated admission rejection details produce a selection failure -(503); if there were no admission rejections, the last unavailable stage produces -a stage-specific 503. Invalid policy signals/configuration or out-of-candidate -picks stop the pass immediately with an internal error. Successful engine -selection ends the pass; forwarding errors do not restart bucket iteration. - -Token ranges and rank belong to the bucket, not its engine groups. For a PD -bucket, both groups share this one request-length decision. Policy fallback on -a cache/affinity miss stays within that group's candidates. There is no second -pass with relaxed admission and no post-policy substitution. - -### Optional SLO ordering - -`Bucket` has optional `ttft_ms` and `tokens_per_second` estimates. The resolver -has independent `ttft_slo` and `tps_slo` preferences: `Disabled` (default), -`SloFirst` (matching first), and `BestEffort` (nonmatching first). The handler -parses `x-sgl-ttft-slo-ms` and `x-sgl-tps-slo` only when their preference is enabled; -invalid enabled headers return 400 before dispatch. Disabled headers are ignored. -SLO targets belong to bucket resolution, not the engine policy's `PickRequest`. - -Absent targets are neutral. A bucket matches TTFT when its positive estimate is -at most the target, and throughput when its finite positive estimate is at least -the target. Missing or invalid estimates do not match a supplied target. Enabled -TTFT targets must be positive; throughput targets must be finite and positive. - -Each unmet preference adds one ordering penalty. With both preferences set to -`SloFirst`, a bucket matching both comes before one matching either, followed by -buckets matching neither. Capacity/rank/ID breaks ties within these tiers. The -same logic applies to plain and PD buckets, and a PD bucket always supplies both -engines. TTFT and throughput preferences never independently resolve P/D groups. -Length constraints are applied first and admission rejection still advances to -the next complete bucket, including a bucket outside the preferred SLO tier. - -Global session modes and sticky policies are follow-ups. Their -integration must preserve bucket-first selection and the same-bucket PD rule. -Cross-bucket affinity probing is not part of this interface. Session/routing -keys still pass through `PickRequest` for policies operating inside the selected -group; unsupported legacy modes need explicit migration decisions before the -standard serving path switches. - -## 4. Policy and admission contracts - -### Policy - -All policies implement one asynchronous, object-safe `Policy::pick` interface. -A boxed future supports the remote prefix indexer; local policies can return an -immediately ready result. - -```rust -pub trait Policy: Send + Sync + std::fmt::Debug { - fn pick<'a>( - &'a self, - engines: &'a [Arc], - request: &'a PickRequest<'a>, - ) -> BoxFuture<'a, Result>; -} -``` - -Policies own their required state handles and read observations locally. Nested -fallback uses `pick_fallback(engines, request)`, which calls the fallback's `pick`; -the fallback reads its own state. There is no shared observation context or cache. -Buckets and HTTP handlers supply only candidates and request facts. - -`Pick` identifies one engine and a selection reason for metrics and tracing. -`PickRequest` carries model, stage, selected bucket ID, input and optional -expected peak counts, optional token IDs, and session/routing keys. It contains -no HTTP body, bucket resolver, state handles, snapshots, or backend configuration. - -| Outcome | Meaning | -| --- | --- | -| `Pick` | The chosen engine belongs to the supplied set and passed admission | -| `NoMatchingBucket` | No bucket supports the requested length | -| `NoCandidates` | No eligible member or policy selection miss | -| `NoAdmissibleEngine` | A policy exhausted its candidates; includes rejection reasons | -| `AdmissionRejected` | The chosen engine failed after-selection admission | -| `InvalidSignal` / `InvalidConfiguration` | Invalid policy input or configuration | -| `OutsideCandidates` | Policy returned an engine outside its exact candidate set | - -### Admission - -Admission evaluates acceptance. It does not rank engines, choose replacements, -change buckets, or mutate affinity. - -An admission policy is a set of per-engine caps, `AdmissionLimits`. Each cap -is optional; an unset cap is not checked, and the default admits everything. -Each cap admits while the engine's load is below it. Usage caps are shares, in -(0, 1], of the engine's reported capacity, so one limit fits engines of any -size. The request itself is not counted: an engine just below a cap still takes -it, and the engine queues or retracts if it overflows. Admission observes load -without reserving it. - -| Limit | Checks | -| --- | --- | -| `max_running_usage` | Running requests against reported max running requests | -| `max_kv_usage` | KV tokens against reported KV capacity | -| `max_waiting_requests` | Reported waiting requests | -| `max_pending_prefill_tokens` | Reported waiting uncached tokens | -| `max_inflight_requests` | Router-local in-flight requests | - -```json -{"max_running_usage": 0.9, "max_kv_usage": 0.95, "max_inflight_requests": 64} -``` - -Unknown fields are rejected during deserialization. - -The policy reads the selected engine's `EngineMetrics` from the load snapshot -it already captured for selection plus the live in-flight counter and calls -`EngineAdmission::check(engine, metrics)`, which returns `Allow`, -`Reject(limit name)`, or an error. Reported -metrics are `None` without a fresh, complete report, never zero, and such -limits fail open; the in-flight count is always known. Policies attach the -checker as `Arc` and decide where checking belongs in -their selection algorithm; there is no placement setting or filtering wrapper. - -Power-of-two first selects an engine, then calls admission exactly once on that -engine. A rejection returns `AdmissionRejected` to the bucket loop; it does not -resample, choose the other sampled engine, or run a policy fallback. No candidates -returns `NoCandidates` without invoking admission. A single candidate is selected -directly; otherwise two distinct candidates are sampled uniformly, and the one -with lower stage pressure wins. A complete tie keeps the first sampled engine. - -Power-of-two reuses the existing pure pressure-comparison functions. Plain and -prefill stages compare estimated prefill queue time when both reports provide it, -then waiting uncached tokens, waiting requests, and running requests. Decode -compares waiting requests, running requests, KV usage fraction, then used KV tokens. -Reported-pressure ties use router-local active requests. If either sampled engine -lacks a fresh, complete native report with valid capacity, both are compared by -router-local active requests instead. Basic reports from older publishers are -still passed to admission when fresh, but do not supply native pressure metrics. - -Power-of-two retains the selected engine's load record from selection and -passes it to admission without another snapshot. A single candidate still has -its load read for admission, even though selection needs no comparison. -Neither the bucket nor HTTP handler supplies observations. Synchronous checks -do not fetch telemetry over the network themselves. - -`AdmissionLimits::default()` is the default for new explicit policy attachments. -It leaves health, role, membership, and policy preferences in force. Migrated -configurations must retain their existing capacity and configured budget checks; -see compatibility below. - -Affinity preferences never bypass attached hard admission. - -Admission checks observe capacity; they do not reserve it. Concurrent requests -may pass against the same observation. Strict reservations would require a -separate mechanism. - -## 5. Concrete policies - -| Policy | Selection behavior | -| --- | --- | -| `RoundRobinPolicy` | Rotate over candidates using a cursor owned by this policy instance | -| `RandomPolicy` | Choose uniformly from candidates | -| `PowerOfTwoPolicy` | Sample two distinct candidates when possible and choose the lower-pressure engine using the stage's load comparison | -| `LeastLoadPolicy` (`load_based`) | Choose the least loaded engine; preserve tie-breaking, telemetry fallback, and recent-dispatch correction | -| `SessionAwarePolicy` | Prefer the session binding; use admitted power-of-two fallback and rebind | -| `StickyPolicy` | Reuse an admitted routing-key binding; use the configured fallback for new or missing keys | -| `CacheAwarePolicy` | Prefer an admitted prefix owner; use power-of-two fallback | - -Session assignments are scoped by model, bucket ID, stage, and session key. -`SessionAwarePolicy::new(store, engine_load)` receives shared state; the caller -owns the store's idle timeout and eviction task. Missing or empty session keys -use power-of-two without creating assignments. A new or out-of-group binding -uses power-of-two with `AdmissionLimits::default()`, then the session policy -checks its selected engine before binding. A concurrent live assignment wins, -but is checked before returning it; rejection ends that attempt without -rewriting the binding or retrying another engine. Existing bindings follow the -shared affinity modes below. Session policies can be attached -independently to each role. Programmatic reorg callers configure -`model.affinity.session_id_header` for HTTP header extraction; this does not -enable legacy global modes or backup escape. - -Session and sticky policies do not create assignments for missing keys. A -binding outside the candidates cannot win. A missing binding may invoke policy -fallback within the group. Rejected session affinity tries fallback without the -old engine; a successful admitted replacement rebinds the session. Failed fallback -preserves the old binding and advances to the next bucket. - -Both affinity policies support `--affinity-mode prefer` (default) and `balanced`. -Prefer retains admissible affinity. Balanced compares it with an admitted -power-of-two alternative and switches only when the `--affinity-balanced-by` load -exceeds both the alternative times `--affinity-load-factor` (default 2) and the -alternative plus `--affinity-load-gap`. The load is `prefill-tokens` (default; -gap default 1024): native waiting uncached tokens plus the request's uncached -tokens on that engine, so the alternative also pays for the prefix the cache owner -holds. Session-aware has no prefix signal and counts the whole input on both -engines. `running-requests` (native or basic reports, gap default 4) ignores the -request. Both reports must be fresh; missing data -or ties preserve affinity. Bindings commit during selection after -admission, as with initial placement; dispatch failure does not roll them back. - -Sticky fallback supports `round_robin`, `random`, `power_of_two`, and `load_based`, -with round-robin as the default. Nested fallbacks use -`AdmissionLimits::default()`; the owning policy explicitly checks the engine -returned by its fallback. - -### Cache-aware behavior - -The architecture preserves the cache algorithm within the supplied candidate -set. Its responsibilities are: - -1. Look up prefix ownership through the local index or remote indexer. -2. Apply minimum matched-token and optional ratio thresholds. -3. Bound candidates using prefix/pressure ordering and the configured minimum, - ratio, and maximum worker counts. -4. Check admission and choose the best remaining prefix holder. -5. Apply the shared affinity mode. With no admitted prefix owner, fall back - within the group, excluding rejected engines. - -Fallback samples two engines and checks the selected engine's admission; it -does not resample on rejection. Candidate limits apply only to prefix selection. - -Memoize the prefix lookup once per request, including remote I/O. Each policy -restricts those matches to its own candidates. A memoized lookup does not imply -that another bucket or stage's cache selection and admission have already run. -Any optimization that skips those steps must establish that the previous result -applies to the current group. - -## 6. Shared state and construction - -Application wiring starts shared services once. Policy construction validates -configuration and passes the required handles to each policy and admission -implementation. For example, `PowerOfTwoPolicy::new(Arc)` -retains the application's shared load table. KV-aware and affinity-aware policies -receive their corresponding shared handles when implemented. Policies with no -state dependency require none. Policy instances do not create duplicate -subscriptions, polling loops, indexes, or remote-client concurrency limits. - -Requirements come from all configured policies, their admission checks, and -nested fallbacks. This includes tokenization, affinity-header extraction, load -observations, and dispatch timestamps. A role-group override must receive its -required settings even when the model's default uses a different policy. - -### Load state - -`state/load_monitor/` owns engine reports and existing router-local request -accounting. Power-of-two owns an `Arc` and captures a snapshot -locally for each nonempty selection attempt. Its `pick` method selects the engine, -then passes that engine's borrowed load record directly to admission. -No snapshot or observation is added to `Pick`, `PickRequest`, -or the bucket interface, and no shared observation context is threaded through -policies or fallbacks. - -The existing snapshot reader preserves rank aggregation, freshness, and capacity -fields. Missing, stale, or rank-incomplete reports yield `None`, not zero load. -A new pick reads current state. Admission reuses the selected observation even if -reports change after selection; it neither recaptures nor reserves capacity. -Fallback policies read their own state and do not share snapshots with callers. - -Snapshot capture still scans the full table; an engine-scoped reader can be added -if profiling justifies it. Power-of-two ranks engines with the shared prefill/decode -comparisons in `state/load_monitor/engine_ranking.rs`, including router-local -fallback. Concrete load-aware admission remains -in #40271. Further shared load interpretation and correction for dispatches since -the report remain follow-ups; these must preserve source, freshness, and available -measurements without adding another -independent in-flight counter. Load and cache observations are not an atomic global -snapshot. Preserve request-guard cleanup. - -### Cache state - -`state/kv_events/` owns local subscriptions, event application, hashing, and the -radix-tree index. The tree stores prefix ownership and storage tiers, not KV -tensors. Eviction, invalidation, and worker removal update this shared state. - -A small `CacheSource` adapter beside the cache policy normalizes local and remote -lookups into common prefix results. Preserve model/cache namespaces, required -rank information, and hash/block-size handling. Convert block counts to token -counts only with a valid conversion. - -A confirmed miss and an unavailable backend both allow cache fallback, but -remain distinguishable in diagnostics. Invalid queries and configuration errors -remain explicit errors. Remote indexing and load-only deployments do not need -a duplicate local cache tree. - -### Affinity state - -`AffinityStore` owns scoped assignments, idle expiry, worker invalidation, and -atomic updates. Policies decide when to reuse or replace an assignment. - -Concurrent first assignments must converge on an effective binding that is -still a candidate and passes admission. If a concurrent binding returns a -different engine, the policy must check that engine before returning it. -Reconcile conflicts with bounded retry. - -Create or replace a binding only after admission succeeds. A binding records -preferred placement, not successful execution, so it may remain if later PD -selection or dispatch fails. It must not increment dispatch accounting. - -## 7. Configuration and compatibility - -Under `--chat-routing reorg`, `--bucket-config` reads a JSON file of complete -buckets. Each bucket sets `plain`, or both `prefill` and `decode`; each group may -set `worker_ids` or `worker_services`, plus `policy`, `admission` and `affinity`. -`affinity` overrides `mode`, `balanced_by`, `load_factor` and `load_gap` for a -session- or cache-aware group; unset fields take the CLI values. - -```json -{ - "ttft_slo": "slo_first", - "buckets": [ - { - "id": "short-context", - "max_input_tokens": 4096, - "max_context_tokens": 8192, - "ttft_ms": 300, - "prefill": {"worker_services": ["inference/prefill-short"], "admission": {"max_pending_prefill_tokens": 32768}}, - "decode": {"worker_services": ["inference/decode-short"], "admission": {"max_kv_usage": 0.9}} - }, - { - "id": "long-context", - "rank": 1, - "max_context_tokens": 131072, - "prefill": {"worker_services": ["inference/prefill-long"]}, - "decode": {"worker_services": ["inference/decode-long"]} - } - ] -} -``` - -A request with 4k input tokens and a 16k expected peak cannot fit the short -bucket's context capacity. It selects the long bucket and both of its P/D groups. -With a known peak of 8k or less, the same input selects both groups of the short bucket. - -Membership has two mutually exclusive forms. `worker_ids` pins exact discovery -identities: URLs for static discovery, `namespace/pod-UID` for Kubernetes pods. -`worker_services` lists Kubernetes `namespace/service` names and matches any -listed Service through its EndpointSlices' `kubernetes.io/service-name` label. -Service membership survives pod replacement and scale-up; worker IDs still -change per pod incarnation so breaker, load and affinity state cannot carry over. -Only Services included by the discovery watch are visible. Multiple slices and -Services selecting one pod contribute the union of its memberships; removing a -slice removes only memberships no longer advertised by another slice. Membership -updates preserve the live worker and its in-flight state. Readiness and routing -use the same group filter, including model, health and role constraints. - -Omitted group fields mean every engine of the group's role and `--policy` -(power-of-two on decode). Admission limits a group leaves unset take -`--max-in-flight` / `--max-kv-usage`, field by field. JSON `null` also inherits; -there is no per-group disable value. To limit only some groups, leave the CLI -default unset and configure those groups explicitly. A group policy is -`power_of_two` or the `--policy` kind, whose affinity, cache and tokenizer -settings are the ones resolved. Startup rejects empty or duplicate IDs, a bucket -that is not exactly plain or P/D, inverted token ranges, a minimum input above -the context capacity, empty membership lists, blank worker IDs, malformed -Service names, both membership fields together, non-positive capacity or SLO -estimates, usages outside (0, 1], zero count limits, and a policy a stage cannot -serve. Without the flag, reorg builds one plain and one P/D bucket -over all engines. The worker registry still rejects mixed plain and PD engines -within one model. - -Legacy `--bucket-config` files describe role-specific buckets and do not -translate: P/D associations must be written explicitly, never inferred from -rank or names. - -Retained settings keep their meanings, defaults, units, and validation unless a -change is listed below. Policy-specific tuning applies to role groups using that -policy. Reject unsupported settings and incompatible combinations at startup; -do not accept and ignore them. - -### Retained behavior - -- Keep `load_based` as the CLI name for `LeastLoadPolicy`. -- Preserve explicit engine membership and context constraints. Bucket-level - ranges and ordering replace independent per-stage selection. - SLO preferences order whole buckets; deployment configuration remains explicit - until the production configuration factory and serving switchover are ready. -- Preserve cache-provider selection, endpoint validation, query timeout and - concurrency limits, and unavailable-backend fallback. -- Preserve the 1,024-token default minimum cache hit, optional ratio gate, - and candidate bounds. Shared affinity modes replace cache diversion knobs. -- Preserve session and sticky headers, idle timeouts, eviction cadence, and the - four sticky fallback choices. Global modes need a bucket-first migration design. -- Map `--filter overloaded` and `--max-in-flight` to `max_inflight_requests`; - the existing router-local counter remains the source. `--max-kv-usage` sets - `max_kv_usage` for groups that leave it unset. -- Preserve configured capacity, pending-prefill, and in-flight checks, including - their missing-report behavior. Power-of-two applies admission to its selected - engine; other policies explicitly place checks in their selection logic. - Other paths use `AdmissionLimits::default()` unless a check is configured. - Never silently discard a configured budget. - -Listener and shutdown configuration, discovery, worker health and circuit -breakers, tokenizer loading, request timeouts, sampling overrides, and logging -remain outside the policy redesign. Preserve their existing behavior and -validation, including dispatch-time breaker probes and request cancellation. - -### Deliberate changes - -| Behavior | Target | -| --- | --- | -| Power-of-two admission | Check only the chosen engine; rejection advances to the next bucket | -| Round-robin cursor | One cursor per role-group policy instance | -| Capacity exhaustion | Try the next compatible bucket; return accumulated rejection details if all fail | -| Primary/backup proposals and post-policy substitution | Removed; each policy returns one engine | -| Session affinity | Reuse or replace bindings through shared affinity modes | -| Omitted `--affinity-mode` | Defaults to `prefer` for both affinity policies | -| Pressure-guard tuning | Replaced by `--affinity-load-factor` and `--affinity-load-gap` | -| Policy attachment | Explicit role-group policy overrides the applicable model/stage default | - -Reject these dropped options explicitly: - -- `--policy fused_score` and `--policy score_policy`, including `--fuse` terms - and weights. -- `--decode-policy legacy_host_affinity`. -- `--stable-pair`. -- `--affinity-mode soft` / `strict`; use `prefer` / `balanced`. -- Legacy pressure guards, cache switch margins, queue limits and saturation floors. -- `--filter prefix_cache` and `--prefix-cache-min-share`. The removed prefix-share - filter is not equivalent to the cache-aware minimum-hit gate. - -## 8. Dispatch and failure handling - -A successful pick means admission passed against the observed state. Health and -capacity can change before dispatch. The handler owns network operations, retry -rules, and accounting for the engines actually dispatched to. - -Selection fallback advances through the ordered compatible buckets before any -network dispatch. Every PD attempt picks both engines from that bucket. With -`--retry-max-attempts` above 1, a dispatch that fails before any response -reaches the client reruns selection from the first bucket with the failed -engine excluded (`BucketRequest::excluded`); it never replaces a pick whose -response the client has started to receive. - -For PD, acquire and release accounting for the actual stages and clean up -partial setup on failure. Policy selection does not own the PD request lifetime. -Selection metrics must not imply that dispatch or execution succeeded. - -## Implementation status - -This PR adds the side-by-side interfaces in `src/buckets_reorg.rs` and -`src/policies_reorg/`, plus a configurable bucket-first implementation behind -`chat_completions`. The live `src/policies/` path remains the default. - -Implemented here: - -- `BucketResolver::resolve` returns all length-compatible buckets in optional - SLO-preference tiers, with capacity/rank/ID order within each tier. -- `Bucket::pick_engines` owns plain/PD orchestration and stage-specific policy - requests; `BucketRequest` carries prepared facts and `BucketPick` retains picks. -- `Bucket` owns input limits, context capacity, rank, and plain-or-PD groups. -- `EngineGroup::pick` owns live candidate filtering, policy invocation, and - exact candidate validation, without cross-bucket fallback. -- `Policy::pick`, within-group fallback interface, per-engine `EngineAdmission::check`, - and `AdmissionLimits` with running and KV usage shares plus waiting, - pending-prefill and in-flight counts. Power-of-two samples two distinct engines, compares stage pressure, - and checks its selected engine with no replacement on rejection. -- Policy-owned load dependency and local observations. Power-of-two passes the - selected engine's load record directly to admission, without another snapshot. - `PickRequest`, `Pick`, and bucket APIs carry no load observations. -- The reorg chat implementation iterates resolved buckets, calls `pick_engines`, - advances on empty candidates/admission rejection, and - dispatches only after one complete selection. Exhaustion retains admission reasons. -- `AppContext::chat_routing` configures legacy versus reorg routing on the same - endpoint and carries the reorg model-resolver map. -- Optional bucket TTFT/throughput estimates and preferences, with enabled-header - parsing and whole-bucket fallback in the shared chat route. -- `CacheAwarePolicy` reads local radix-tree or remote indexer prefixes, intersects - exact worker URLs with the current group, applies hit thresholds and candidate - bounds, and applies the shared affinity mode. -- `PrefixMemo` shares lookup results (including misses and unavailable backends) - across bucket attempts for one prepared request. Entries are keyed by the shared - `Arc` so different index namespaces remain independent. Each pick - reruns its own candidate filtering and admission after obtaining a fresh snapshot. -- Cache selection checks bounded candidates explicitly; rejected owners are - excluded from fallback. The cache policy checks its fallback winner's admission. - Cache policies require plain/prefill groups. -- `SessionAwarePolicy` reuses admitted model/bucket/role-scoped bindings from a - shared `AffinityStore`, falling back to power-of-two for new or keyless sessions. - Assignments follow admission; concurrent binding winners are rechecked. - Rejected affinity invokes fallback; failed fallback preserves the binding. - The caller owns expiry and sweeper lifecycle. A binding may remain after a - later PD group fails, because it records placement rather than dispatch. - -- `--bucket-config` builds explicit plain and P/D buckets with per-group - membership, policy and admission; omitting it builds the default buckets. - -Follow-up work includes remaining selection policies. - -Not yet implemented in the reorg path: - -- Other concrete selection policies. -- Global session modes and sticky routing-key affinity. -- Power-of-k cache-miss fallback configuration and cache decision metrics. -- Shared load interpretation, dispatch correction, and policy-specific - dispatch-timestamp requirements. -- PD compatibility filtering and legacy-route switchover. - -The preceding policy sections describe target behavior for those follow-ups; -they do not claim those capabilities are present in this PR. diff --git a/experimental/sgl-router/README.md b/experimental/sgl-router/README.md index ccb1305e09..50ed7b9839 100644 --- a/experimental/sgl-router/README.md +++ b/experimental/sgl-router/README.md @@ -1,738 +1,353 @@ # sgl-router -Slim, KV-aware, OpenAI-compatible router for SGLang workers. +Slim, KV-aware router for SGLang workers. -Serves a single model and routes across its workers. Exposes -`/v1/tokenize`, `/v1/detokenize`, `/v1/models`, [`/v1/embeddings`](#embeddings), -[`/v1/classify`](#classify), [`/v1/rerank`](#rerank), -[`/v1/chat/completions` and `/v1/completions`](#openai-over-generate) and SGLang's native -[`/generate`](#native-generate) (buffered and SSE), plus -`/healthz` / `/readyz` and `/metrics`. Worker pools come from either a static URL -list or Kubernetes EndpointSlice discovery. Both edges speak cleartext HTTP/2 -where the peer does — see [HTTP/2](#http2). +A router serves one model and spreads its requests across that model's SGLang +engines. It picks an engine per request from live engine load, global KV-cache prefix +matches, or session affinity, supports prefill/decode (P/D) disaggregation, bucketed deployments, and +discovers engines from a static URL list or Kubernetes EndpointSlices. -## Building +- [Architecture](#architecture) +- [Quick start](#quick-start) +- [Key interfaces](#key-interfaces) +- [Routing policies](#routing-policies) +- [Deployment examples](#deployment-examples) +- [Add a new policy](#add-a-new-policy) +- [Other features](#other-features) + +Related docs: [ARCHITECTURE.md](ARCHITECTURE.md) (routing design and bucket +configuration), [REFERENCE.md](REFERENCE.md) (detailed feature behavior), +[monitoring/README.md](monitoring/README.md) (metrics and +Grafana), [BENCHMARKS.md](BENCHMARKS.md), +[sgl-kv-indexer/README.md](sgl-kv-indexer/README.md) (external KV indexer). + +## Architecture + +A deployment is a **router** in front of **buckets of SGLang engines**. A bucket +is a logical group defined in the router configuration. It holds either one +**plain** engine group, where each engine does prefill and decode, or separate +**prefill and decode (P/D)** groups. Buckets can be length tiers (for example +short and long context); without a bucket file there is one plain bucket and one +P/D bucket containing every discovered engine. + +![Router deployment with short-context and long-context P/D engine buckets](./assets/architecture-deployment.png) + +Engines publish load reports and KV events. The router uses them, plus its own +in-flight counts and session bindings, to pick engines. An optional +[external KV indexer](sgl-kv-indexer/README.md) can replace the router's local +radix tree as the source of prefix matches. In P/D, the router sends the request +to both the chosen prefill and decode engines, and prefill transfers KV directly +to decode. + +For each request, the router renders and tokenizes it, tries the buckets that fit +its length (ordered by SLO preference, capacity, then `rank`), lets the group's +policy pick an engine that passes admission, and dispatches it, retrying on +another engine if it fails before any response. Output handling (OpenAI +responses, tool and reasoning parsing) also happens in the router. + +**Scalability.** Routers are stateless: nothing is persisted or shared between +replicas. Each one rebuilds its routing state (engine load, KV prefix tree, +in-flight counts) from the engines' own reports, so you scale a large deployment +by running more router replicas behind a load balancer. A new replica can copy +the KV tree from a warm sibling at startup +([peer bootstrap](#kv-cache-signals)). Session bindings stay in each +replica's memory, so with `session_aware` and several replicas, route a session's +requests to the same replica (for example, a load balancer sticky on the session +header). + +## Quick start + +Build: ```bash cd experimental/sgl-router cargo build --release ``` -## Running - -The router is configured entirely through CLI flags (run -`sgl-router --help` for the full list). It serves exactly one model, so -`--model-id` is required, along with exactly one discovery backend. -`--tokenizer-path` is optional: give it a local `tokenizer.json` path or a -HuggingFace repo id, and when omitted the router downloads the tokenizer -for `--model-id` from HuggingFace (honoring `HF_TOKEN` / `HF_HOME`). - -Static worker list: +Run in front of two engines with cache-aware routing: ```bash -sgl-router \ +./target/release/sgl-router \ --host 0.0.0.0 --port 30000 \ --model-id qwen3 \ - --tokenizer-path /models/qwen3/tokenizer.json \ - --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 -``` - -Kubernetes EndpointSlice discovery: - -```bash -sgl-router \ - --host 0.0.0.0 --port 30000 \ - --model-id qwen3 \ - --tokenizer-path /models/qwen3/tokenizer.json \ - --service-discovery \ - --service-discovery-namespace prod \ - --selector app=engines-qwen3 -``` - -Omit `--service-discovery-namespace` to watch all namespaces (requires -cluster-wide RBAC). For prefill/decode disaggregation, replace `--selector` -with `--prefill-selector` and `--decode-selector`. - -To run two engine versions side by side in PD mode (e.g. during a rollout), -pass `--pd-version-group-label `. A prefill worker is then paired only with -decode workers that have the same value for that EndpointSlice label (inherited -from the Service, so use one Service per role and version), so KV never crosses -versions. Unlabeled workers form their own group. - -```bash -sgl-router --model-id qwen3 --service-discovery \ - --prefill-selector app=engines-qwen3,role=prefill \ - --decode-selector app=engines-qwen3,role=decode \ - --pd-version-group-label sglang.ai/version-group -``` - -External KV indexer as the cache-aware signal source: - -```bash -sgl-router \ - --model-id qwen3 \ - --tokenizer-path /models/qwen3/tokenizer.json \ --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \ - --policy cache_aware \ - --cache-prefix-provider indexer \ - --kv-indexer-endpoint http://10.0.0.10:50051 \ - --kv-indexer-query-timeout-ms 100 \ - --kv-indexer-query-max-inflight 32 -``` - -The Indexer replaces the Router-local radix tree as the native Cache-Aware -signal. Query timeouts and local concurrency are bounded by the two Indexer -options, which default to 100 ms and 32 respectively. - -### LoRA cache identity - -LoRA routing requires workers that publish `BlockStored.lora_name`. Requests -select the adapter through `model: "base:adapter"` on OpenAI routes, or -`lora_path` on native routes. Salted generation requests also carry `cache_salt`. - -Worker block IDs are opaque and may differ across adapter loads or workers. -The router derives adapter-aware lookup keys from event tokens and keeps a -per-worker, per-DP-rank mapping for parent links and removals. Multiple block -IDs can own the same lookup key; removing one leaves the other owners and -storage tiers intact. The remote indexer bridge performs the same translation. -Base-model events retain their existing published-key path. - -New replicas request `?include_lora=true` snapshots containing LoRA mappings -(version 2). Legacy consumers receive a base-only version-1 view, so mixed -snapshots do not prevent base-cache bootstrap during rolling upgrades. Base-only -snapshots retain version 1 for every consumer. LoRA stores buffered during bootstrap retain their -tokens; the original base-model buffering and replay behavior is unchanged. - -Reloading different adapter weights under the same public name can still -credit stale engine blocks until their eviction. Names identify adapters; -versioned names are needed to distinguish different weights. - -### KV-event gap replay - -ZMQ drops KV events at the publisher's high-water mark, and a lost removal -leaves the router crediting a worker with blocks it no longer holds. Give each -engine a replay socket and the router re-fetches the batches a sequence gap -skipped: - -```bash -python -m sglang.launch_server ... \ - --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558"}' -``` - -Rank r publishes on 5557 + r and replays on 5558 + r, so with `dp_size > 1` -place the replay base at least dp_size ports past the PUB base. The engine keeps -the last `buffer_steps` batches (default 10000); older gaps stay unrepaired. -`sgl_router_kv_event_replays_total{outcome}` counts the attempts. - -### Pending prefixes - -Until the engine's KV events arrive, requests that share a cold prefix see no -cached owner and spread across workers, each prefilling the same prefix. This -is common with parallel sampling, RL rollouts and agent fan-out. -`--cache-pending-prefix-ttl-ms` credits a worker with a prompt's prefix for that -long after routing it there, so the burst stays together. It is off by default -and needs `--policy cache_aware` with the Router-local radix tree. -`sgl_router_cache_pending_prefix_hits_total` counts lookups where a pending -prefix matched deeper than any confirmed one; the policy's candidate and -admission filters still decide whether that worker is picked. - -### Peer bootstrap (Kubernetes) - -A replica that starts mid-fleet subscribes to each worker's KV topic -mid-stream, so everything already resident in the engines' caches is invisible -to it and it routes cache-blind until traffic re-stores those blocks — -degrading the engines' locality for the whole fleet, once per replica on every -rolling update. With a peer selector set, a booting replica instead pulls a -tree snapshot from a warm sibling over `/internal/kv_snapshot` and splices it -under its own live delta stream, and `/readyz` stays 503 until that bootstrap -settles. Off unless `--kv-peer-selector` is set; requires `--policy -cache_aware` with the Router-local radix tree (not an external Indexer) and -`--service-discovery`, which the CLI enforces. - -```bash -sgl-router \ - --model-id qwen3 --policy cache_aware \ - --service-discovery --service-discovery-namespace prod \ - --selector app=engines-qwen3 \ - --kv-peer-selector kubernetes.io/service-name=sgl-router -``` - -- `--kv-peer-selector` is matched against **EndpointSlice labels** — the - router Service's labels plus `kubernetes.io/service-name`, never the pods' - labels, so `kubernetes.io/service-name=` is the unambiguous - choice. A pod-template label matches nothing and every replica boots cold. -- `--kv-bootstrap-timeout-ms` (default 600000) bounds the whole bootstrap; - readiness waits on it, so startup/readiness probes must tolerate a replica - staying unready this long. -- `--kv-bootstrap-fetch-timeout-cap-ms` (default 300000) caps one snapshot - fetch within that budget. -- `--kv-bootstrap-seed-required` (off by default) keeps holding `/readyz` when - siblings were present but their tree could not be pulled, bounded at - max(3× the bootstrap timeout, 60s) — it delays a failed seed's replica (and - with it a rolling update) rather than shipping it cache-blind; it cannot - stall a rollout indefinitely. - -The deployment needs two things beyond the flags: the router's ServiceAccount -must hold `get`/`list`/`watch` on `endpointslices` in the watched namespace -(the worker-discovery Role already grants this), and the pod spec should wire -`POD_NAME`, `POD_NAMESPACE` and `POD_IP` from the downward API so a replica -can exclude itself from its own peer list. A rolling update should surge -(`maxUnavailable: 0`) so new replicas always have warm siblings to copy from. -`tests/e2e/k8s_integration/manifests/kv-bootstrap.yaml` is a worked example of -all of it, and the `sgl_router_kv_bootstrap_*` series in -[monitoring/README.md](monitoring/README.md) make the whole path observable. - -### Reorg routing - -Use `--chat-routing reorg` to select the new bucket engine. The existing `--policy` -and cache/session flags configure its policies; a bucket file is optional. - -```bash -sgl-router --model-id qwen3 --worker-urls http://localhost:30001 \ --chat-routing reorg --policy cache_aware ``` -Reorg supports `power_of_two` (its default), `cache_aware`, and `session_aware`. -Discovery supplies the plain or PD workers; decode uses power-of-two. Cache -settings, external indexers and session headers/timeouts retain their existing -flags. Admission flags set the default per-engine caps; an engine at a cap is -skipped, and affinity falls back to the others: +Send a request: + +```bash +curl http://localhost:30000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model": "qwen3", "messages": [{"role": "user", "content": "Hello"}]}' +``` + +`--model-id` is required. `--tokenizer-path` takes a local `tokenizer.json` or a +HuggingFace repo id; when omitted, the tokenizer for `--model-id` is downloaded +from HuggingFace (honoring `HF_TOKEN` / `HF_HOME`). `sgl-router --help` lists +every flag. + +## Key interfaces + +### HTTP endpoints + +| Endpoint | Purpose | +| --- | --- | +| `POST /v1/chat/completions`, `POST /v1/completions` | OpenAI API, served [over `/generate`](#openai-over-generate) | +| `POST`/`PUT /generate` | SGLang [native generate](#native-generate), buffered or SSE | +| `POST /v1/messages`, `/v1/messages/count_tokens` | [Anthropic Messages](#anthropic-messages) | +| `POST /v1/embeddings`, `/v1/classify`, `/v1/rerank` | [Embeddings, classify and rerank](#embeddings-classify-and-rerank) | +| `POST /v1/tokenize`, `/v1/detokenize` | Tokenize with the router's tokenizer | +| `GET /v1/models` | The served model | +| `POST /flush_cache` | Fan out a cache flush to every worker | +| `GET /healthz`, `/readyz` | Liveness and readiness | +| `GET /metrics` | Prometheus metrics, see [monitoring/README.md](monitoring/README.md) | + +Both edges speak cleartext HTTP/2 where the peer does; see +[operations](#operations). With `--grpc-port`, the router also serves the +engine's [native gRPC](#native-grpc) API. + +### Key flags + +| Area | Flags | +| --- | --- | +| Model | `--model-id`, `--tokenizer-path`, `--no-tokenizer` | +| Engines | `--worker-urls`, or `--service-discovery` with `--selector` (plain) or `--prefill-selector` + `--decode-selector` (P/D) | +| Routing | `--chat-routing reorg`, `--policy`, `--bucket-config`, `--affinity-*`, `--session-id-header` | +| Admission | `--max-in-flight`, `--max-waiting-requests`, `--max-pending-prefill-tokens`, `--max-running-usage`, `--max-kv-usage` | +| Reliability | `--retry-max-attempts`, `--request-timeout-secs`, `--stale-request-timeout-secs` | +| KV signals | `--cache-prefix-provider`, `--kv-indexer-endpoint`, `--kv-peer-selector` | + +Omitting `--chat-routing` keeps the legacy routing path (default +`--policy round_robin`). This README and [ARCHITECTURE.md](ARCHITECTURE.md) +describe `--chat-routing reorg`. + +## Routing policies + +Each engine group runs one policy, chosen by `--policy` (default `power_of_two` +under reorg) or per group in a bucket file: + +- **`power_of_two`**: samples two engines and takes the less loaded. Prefill and + plain engines compare estimated prefill queue time, waiting uncached tokens, + waiting, running, then in-flight requests; decode engines compare waiting and + running requests, then KV usage. Decode groups always default to it. +- **`cache_aware`**: prefers the engine holding the longest cached prefix of the + prompt (at least `--cache-affinity-min-matched-tokens`, default 1024, and + optionally `--cache-affinity-min-match-ratio` of the prompt). Plain and + prefill groups only. Needs a tokenizer. +- **`session_aware`**: keeps requests with the same session header + (`--session-id-header`, default `x-session-id`) on the engine they are bound + to. Any stage, including decode via a bucket file. + +Both affinity policies fall back to `power_of_two` among the remaining engines +when there is no preferred engine or it fails admission. With +`--affinity-mode prefer` (default) the preferred engine wins whenever it is +admitted. With `--affinity-mode balanced` the router also samples an +alternative and switches only when the preferred engine's load exceeds both +`alternative * --affinity-load-factor` (default 2) and +`alternative + --affinity-load-gap`, measured by `--affinity-balanced-by` +(`prefill-tokens` by default, or `running-requests`). + +Admission flags set per-engine caps; an engine at a cap is skipped: | Flag | Rejects an engine at | |---|---| -| `--max-in-flight N` | `N` requests this router has in flight to it (no `--filter overloaded` needed) | +| `--max-in-flight N` | `N` requests this router has in flight to it | | `--max-waiting-requests N` | `N` engine-reported waiting requests | | `--max-pending-prefill-tokens N` | `N` engine-reported waiting uncached tokens | | `--max-running-usage F` | share `F` of its reported running-request capacity | | `--max-kv-usage F` | share `F` of its reported KV capacity | -Engine-reported caps lag by one load report, so a burst can pass them; the -router-local `--max-in-flight` counts dispatches as they happen. -Unsupported legacy options fail at startup. +Engine-reported caps lag by one load report, so a burst can pass them; +`--max-in-flight` counts dispatches as they happen. -`--bucket-config buckets.json` replaces the default plain and P/D buckets. Each -bucket is plain or P/D, and each group may set its own engines, policy, -admission and affinity; see [POLICY_DESIGN.md](POLICY_DESIGN.md#7-configuration-and-compatibility): +For the full selection rules see ARCHITECTURE.md: +[load-aware routing](ARCHITECTURE.md#13-example-load-aware-routing), +[cache- and session-aware routing](ARCHITECTURE.md#14-example-cache-aware-and-session-aware-routing), +and [configuration examples](ARCHITECTURE.md#2-configuration-examples). -```json -{"buckets": [{ - "id": "default", - "prefill": {"worker_services": ["inference/prefill"], "admission": {"max_pending_prefill_tokens": 32768}}, - "decode": {"worker_services": ["inference/decode"], "admission": {"max_kv_usage": 0.9}} -}]} -``` +## Deployment examples -`worker_services` matches Kubernetes Services by `namespace/name`, using the -`kubernetes.io/service-name` label on watched EndpointSlices. Replacement pods -and new replicas join the same group automatically. The router's discovery -selectors must include those EndpointSlices; this field does not expand the watch. -A worker selected by several Services belongs to each of them. - -For static URL discovery, use `"worker_ids": ["http://worker:30000"]`: each ID -is the configured worker URL. Kubernetes worker IDs are `namespace/pod-UID` -(and change when a pod is replaced), so use `worker_services` for durable pools. -Set only one membership field, or omit both for every engine of the group's role. -Empty membership lists and blank worker IDs are rejected at startup. - -Admission fields left unset or set to `null` inherit CLI defaults. To apply a -limit only to selected groups, omit that CLI default and set it on those groups. - -Omitting `--chat-routing` keeps the existing policies and defaults. - -Both reorg affinity policies accept `--affinity-mode prefer` (default) or -`balanced`. Prefer keeps an admissible session binding or the best admissible -prefix owner. Balanced samples a power-of-two alternative and switches only -when the affinity engine's load exceeds both -`alternative * --affinity-load-factor` (default 2) and -`alternative + --affinity-load-gap`. `--affinity-balanced-by` picks the load: -`prefill-tokens` (default; gap default 1024) is the engine's waiting uncached -tokens plus the prompt tokens it would prefill for this request, so a cache owner -is credited for its prefix (session-aware assumes the whole prompt on either -engine); `running-requests` (gap default 4) ignores the request. Missing fresh load preserves admissible -affinity; ties also preserve it. - -A bucket group may override these with `"affinity": {"mode", "balanced_by", -"load_factor", "load_gap"}`; unset fields take the CLI values, and the balanced -fields require `"mode": "balanced"`. For example, a session-aware group balanced -by running requests: -`"plain": {"affinity": {"mode": "balanced", "balanced_by": "running_requests"}}`. - -Both modes fall back within the group when affinity fails admission, excluding -rejected engines. The fallback winner must pass admission; failure advances to -the next bucket. Session replacements are bound after admission during selection, -not after dispatch. Reorg rejects legacy pressure guards, cache switch margins, -and queue/saturation gates in favor of these shared affinity settings. - -### Optional tokenizer for load-only routing - -`--no-tokenizer` skips tokenizer loading for load-only policies such as -`power_of_two` and `session_aware`, on either routing path. Workers tokenize the -original messages, and `/v1/tokenize` and `/v1/detokenize` are unavailable. -Cache-aware routing, prefix-cache terms or filters, and buckets with token-length -or context limits require a tokenizer. Reorg buckets that only select membership -and load-based policies can use `--no-tokenizer`. - -### DP-rank routing - -An engine launched with `--dp-size` or `--attn-dp-size` runs several DP ranks, -each with its own KV cache, behind one endpoint. With `--dp-aware`, the router -also picks the rank inside the selected worker. It sends that rank as -`X-Data-Parallel-Rank`, which the engine honors, and overwrites any value the -client sent. The router picks the first of these that applies: - -1. A hash of the sticky routing key or session id, so a conversation keeps - its rank and router replicas agree. -2. The rank with the deepest cached prefix in the local KV tree. -3. The rank with the fewest requests this router has in flight on it, or with - `--dp-rank-policy round_robin`, the worker's ranks in turn. - -In PD mode, decode is ranked by step 3 only. The decode request names the -prefill rank in `disagg_prefill_dp_rank`, and the bootstrap room satisfies -`room % prefill_dp_size == prefill_rank`. - -### Retries - -`--retry-max-attempts N` (default 1, which disables retries; 3 is typical) lets -a request that fails before any response reaches the client be sent again to a -worker it has not tried yet. A failure is a transport error, an open circuit -breaker, a 5xx, a 429, or a timeout: `--request-timeout-secs` bounds a whole -non-streaming response, and a streaming one until its headers arrive. SGLang's -chat endpoint sends those headers with the first token, so for streaming chat -the same flag bounds time to first token, queueing included; raise it for -long-context or deeply queued fleets. Once a streaming response's 2xx status -has been sent, it is never retried, and neither is any other 2xx or 4xx. In PD mode the failed -side is excluded, or both sides when the caller set `rid`, which an engine still -running it would refuse; a new pair gets a new bootstrap room. Retries share the -request's `--stale-request-timeout-secs` deadline, and none starts after it. When -every eligible worker has failed, the client gets the last failure. -`sgl_router_retries_total` counts the retried attempts. - -Each retry first waits out a jittered exponential backoff: a random delay in -`[d/2, d]`, where `d` starts at `--retry-initial-backoff-ms` (default 50), -doubles per retry, and is capped at `--retry-max-backoff-ms` (default 2000). -Set the initial backoff to 0 to retry immediately. A backoff never waits past -the request's stale deadline. - -### Engines with `--api-key` - -The router reads each worker's `/server_info` and `/model_info` to learn its -model, KV-event publisher, HTTP/2 support and DP size. An engine launched with -`--api-key` rejects those requests without the key, so pass the same key as -`--worker-api-key`. Chat requests and `/flush_cache` forward the caller's -`Authorization` header instead, so callers still need the engine key. - -### Fleet-wide sampling contract - -`--override-sampling-params` fixes the sampling configuration for every client -of this router, independently of what the engine's own defaults happen to be -(on native [`/generate`](#native-generate), in each prompt's `sampling_params`): +Static workers, plain engines: ```bash -sgl-router \ - --model-id qwen3 \ - --tokenizer-path /models/qwen3/tokenizer.json \ - --worker-urls http://10.0.0.1:30000 \ - --override-sampling-params '{"temperature": 1, "top_p": 0.95, "n": 1}' \ - --sampling-param-conflict reject +sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \ + --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 ``` -It takes one JSON object keyed by the request-body field names -(`temperature`, `top_p`, `top_k`, `min_p`, `repetition_penalty`, -`frequency_penalty`, `presence_penalty`, `n`). A configured value is injected -whenever the request omits that field, so the engine's defaults cannot drift -from what the operator declared. `temperature`, `top_p`, `top_k`, `min_p` and -`repetition_penalty` are the five the engine resolves from the model's own -`generation_config`, which is what makes them drift when a deployed image -changes; the rest have fixed API defaults. +Kubernetes discovery, plain engines (omit the namespace to watch all +namespaces, which needs cluster-wide RBAC): -An explicit `null` counts as omitting the field, not as a client-supplied -value: the OpenAI API types these parameters as nullable with a documented -default, so `null` asks for the default — and on a governed fleet the -configured value is what the default is. +```bash +sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \ + --service-discovery --service-discovery-namespace prod \ + --selector app=engines-qwen3 +``` -For a request that does send a value, `--sampling-param-conflict` decides: -`reject` (the default) 400s a differing value before admission, while `allow` -forwards the client's value untouched — the router never silently rewrites what -a client sent. A `reject` response carries -`x-router-error-code: sampling_contract_violation` and is counted in -`sgl_router_sampling_contract_rejections_total{param}`, so a rollout's blast -radius is visible per parameter rather than folded into `bad_request`. +Kubernetes discovery, P/D engines with session affinity: -`reject` also 400s a value it cannot read as a number, on a governed parameter -only. The engine coerces more than JSON numbers — a bool and a numeric string -both become numbers — by rules that are undocumented and need not match across -a fleet, so a value the router cannot read is one it cannot prove conforms, and -waving it through would make the pin bypassable. The common coercions are -matched exactly (`false` is 0, `"0.5"` and `"1_0"` are 0.5 and 10), so this -refuses only genuine garbage. `allow` is unaffected: it promises nothing, so -such a value keeps flowing to the engine, which owns the request schema. +```bash +sgl-router --model-id qwen3 --chat-routing reorg --policy session_aware \ + --service-discovery --service-discovery-namespace prod \ + --prefill-selector app=engines-qwen3,role=prefill \ + --decode-selector app=engines-qwen3,role=decode \ + --max-kv-usage 0.9 +``` -A value may also be an inclusive band, `{"min": LO, "max": HI}`, for a -parameter that stays tunable inside a range. A band names no value to inject, -so it constrains only the requests that name the parameter; one that omits it -gets the model's own `generation_config` default, which the router cannot see. -Because a band can only ever reject, combining one with `allow` is a startup -error. +Length-tiered buckets from a file (schema and a full example in +[ARCHITECTURE.md](ARCHITECTURE.md#example-c-pd-buckets-from-a-file)): -Values are range-checked at startup, so a misconfiguration fails the launch -instead of 400ing every request at the engine. Repeating a key in the flag is -also a startup error, rather than silently enforcing whichever copy came last. +```bash +sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \ + --service-discovery --service-discovery-namespace prod \ + --prefill-selector app=engines-qwen3,role=prefill \ + --decode-selector app=engines-qwen3,role=decode \ + --bucket-config buckets.json +``` -| parameter | accepted | notes | -| --- | --- | --- | -| `temperature` | `[0, 2]` | | -| `top_p` | `(0, 1]` | | -| `top_k` | `>= 1`, or exactly `-1` | `-1` is the engine's "whole vocabulary" spelling and its default. Being non-contiguous it cannot bound a band. Note `top_k: 1` is greedy decoding, not "disabled". | -| `min_p` | `[0, 1]` | not an OpenAI parameter; the engine's domain | -| `repetition_penalty` | `(0, 2]` | not an OpenAI parameter; the engine's domain | -| `frequency_penalty` | `[-2, 2]` | | -| `presence_penalty` | `[-2, 2]` | | -| `n` | `[1, 128]` | | +## Add a new policy -The OpenAI domains are deliberately narrower than what the engine itself -accepts (`SamplingParams.verify` would take `temperature: 5`): these values are -injected into request bodies, and a fleet contract outside the range every -OpenAI client library validates against is far more likely a typo than an -intent. +Reorg policies live in [`src/policies_reorg/`](src/policies_reorg/). Contributions +are welcome; read [ARCHITECTURE.md](ARCHITECTURE.md) first for how buckets, +admission and fallback fit together. -Cost: a request that named every configured field forwards its original bytes -untouched. One that omits a field has the scalars spliced directly into the -body bytes — no parse, no re-serialize — which on a 16 MiB body is ~0.24 ms -against ~5.7 ms for a `serde_json` round-trip. Only `input_ids` and PD -bootstrap injection still parse and re-serialize, because those may have to -overwrite a key the client sent. +1. **Implement the `Policy` trait** in `src/policies_reorg/.rs`. The trait + is in [`src/policies_reorg/mod.rs`](src/policies_reorg/mod.rs): -### Relationship to the engine's own `--preferred-sampling-params` + ```rust + pub trait Policy: Send + Sync + Debug { + fn pick<'a>(&'a self, engines: &'a [Arc], request: &'a PickRequest<'a>) + -> BoxFuture<'a, Result>; + fn supports(&self, _stage: Stage) -> bool { true } + fn needs_request_tokens(&self) -> bool { false } + fn fallback(&self) -> Option<&dyn Policy> { None } + } + ``` -The engine has an inject-when-absent flag of its own, -`--preferred-sampling-params`, merged in -`python/sglang/srt/managers/tokenizer_manager.py` as -`{**preferred, **obj.sampling_params}`. On `/v1/chat/completions` it is -currently a no-op: `ChatCompletionRequest.to_sampling_params` -(`python/sglang/srt/entrypoints/openai/protocol.py`) resolves every sampling -key eagerly through `generation_config` and then its own defaults, so the -right-hand side of that merge is always fully populated and always wins. -(`/v1/responses`, in the same file, already omits `None` entries for exactly -this reason.) If that is fixed engine-side, `--preferred-sampling-params` -covers the inject-when-absent half for a single engine. + - `pick` must return one of the supplied `engines` and check admission on it + (`self.admission.check(&engine, &metrics)`), returning + `PickError::AdmissionRejected` on rejection. + - The policy holds its own shared-state handles (load table, KV source, + session store), passed in at construction, and reads them inside `pick`. + - Override `supports` to restrict stages (plain, prefill, decode), + `needs_request_tokens` if it reads `request.token_ids` or `request.prefix`, + and `fallback` if it delegates to another policy. + - [`power_of_two.rs`](src/policies_reorg/power_of_two.rs) is a complete + ~80-line example; [`session_aware.rs`](src/policies_reorg/session_aware.rs) + shows affinity with a fallback. +2. **Declare the module** in `src/policies_reorg/mod.rs`. +3. **Add a `PolicyKind` variant** in [`src/config/types.rs`](src/config/types.rs) + with its `--policy` name. `PolicyKind` is shared with the legacy path, so add + an arm returning an error to `build_kind` in + [`src/policies/factory.rs`](src/policies/factory.rs) (the compiler lists every + exhaustive match to update). +4. **Wire it into** [`src/policies_reorg/factory.rs`](src/policies_reorg/factory.rs): + add it to the allowlist in `validate()` and construct it in the `match kind` + of `Groups::build`. +5. **Test it** in `tests/component/policies_reorg_.rs`, registered in + `tests/component/main.rs`, following `policies_reorg_power_of_two.rs`. For + end-to-end routing, add a case under `tests/proxy/chat_routing/reorg/`. +6. **Document it**: a subsection in [ARCHITECTURE.md](ARCHITECTURE.md) next to + section 1.4, and a bullet in [Routing policies](#routing-policies) above. -What stays the router's to own either way is the enforcement half: the -`reject` immutability contract with its 400 before admission — costing no -queue slot and no engine round-trip — the `sampling_contract_violation` code -and per-parameter counter, bands, and one contract applied at a shared ingress -across engines whose own flags the router operator may not control. +## Other features -## Chat rendering +Each item links to its full description in [REFERENCE.md](REFERENCE.md). -The router renders chat requests with dynamo-render (`dynamo-renderer`): the model's -HF Jinja template from `tokenizer_config.json` or a sibling -`chat_template.jinja`, or dynamo-render's built-in DeepSeek encoder (V4 family, V3.2) -for template-less models. Cache-aware routing hashes the rendered tokens so its -prefix queries match the blocks the engine caches. Models the engine encodes in -code but dynamo-render cannot tokenize here (Inkling) route via raw prompt -text, as does any model whose template fails to load or render. +### Native `/generate` -Some chats additionally forward the rendered tokens to the engine as -`input_ids`, retaining the original messages, so the engine skips -re-tokenizing. How many depends on the model's renderer. DeepSeek-V4's, V4.1's -and Kimi-K3's native encoders and GLM-5.3's and Qwen3.8's templates are -fixture-verified against SGLang's request normalization, so they forward every -chat except multimodal ones and those with caller-provided `input_ids`. Renderers -without that verification (HF Jinja templates) -forward only plain text chat requests (string `content`, no tools, no template -kwargs or reasoning controls or historical `reasoning_content`, no assistant -continuation, no consecutive users or non-leading system turns) and warn -`UNVERIFIED` at startup; every other request shape is rendered for routing -only. +Same body and response as the engine's `/generate`, buffered or SSE. With a +tokenizer loaded, `text` is forwarded as `input_ids` so routing sees the +engine's exact tokens (`--disable-input-ids-forwarding` turns this off). See +[native `/generate`](REFERENCE.md#native-generate). -Use matching model files on the router and workers, and set -the same `--default-chat-template-kwargs`, `SGLANG_DEFAULT_THINKING`, -`SGLANG_DSV4_REASONING_EFFORT`, and `SGLANG_DSV41_REASONING_EFFORT` on both. -The router reads these render defaults from its own configuration and environment; -it does not discover the workers' settings. Point `--tokenizer-path` at the workers' -model snapshot so the V4 effort profile is read from the same -`encoding/encoding_dsv4.py`. +### OpenAI over /generate -Set `--disable-input-ids-forwarding` for this router's model when worker-side -rendering has not been verified to match. This disables router-generated IDs -for every routing policy; cache-aware routing still renders and tokenizes -locally, and the original messages reach the workers for engine processing. -Caller-supplied `input_ids` remain caller-owned and pass through unchanged. +For `/v1/chat/completions` and `/v1/completions`, the router runs SGLang's +OpenAI layer itself: it sends the request to `/generate` and builds the OpenAI +response. Requests it cannot reproduce exactly (for example media or strict +tools) go to the engine's own route unchanged. See +[OpenAI over /generate](REFERENCE.md#openai-over-generate) for the exact +conditions, and [chat rendering](REFERENCE.md#chat-rendering) plus the +per-model sections ([DeepSeek V4 and V4.1](REFERENCE.md#deepseek-v4-and-v41), +[GLM-5.3](REFERENCE.md#glm-53), [Qwen3.8](REFERENCE.md#qwen38), +[Kimi-K3](REFERENCE.md#kimi-k3)) for which chats forward `input_ids`. -Forwarding logs its assumptions at startup. In particular, disable it when the -router's render defaults differ from the workers', for worker template overrides -not reflected in the router's model files, for worker parser overrides such as -`--tool-call-parser deepseekv32` that select a native encoder over a shipped -template, or conversation templates with stop strings (the engine's `input_ids` -path skips those template stops). Disabling forwarding preserves engine behavior -but does not establish parity for local routing hashes. +### Anthropic Messages -Also set `--disable-input-ids-forwarding` for array-only templates: Dynamo may wrap -string content into arrays differently from the worker. The pinned Dynamo renderer does not expose -its conversion flag, so the router cannot automatically block these templates. -Detailed content-format parity coverage follows in #39133. +`/v1/messages` is converted to a chat completion and back. +`/v1/messages/count_tokens` is answered by the router from the chat template. +See [Anthropic Messages](REFERENCE.md#anthropic-messages). -The Dynamo crates are pinned exactly and `Cargo.lock` is committed; CI builds -with `--locked`, so rendered bytes cannot change without a reviewed diff. +### Embeddings, classify and rerank -Router tokenization sits on the TTFT path for every chat it renders. Two opt-in -flags make it cheaper: +Engine request and response bodies, routed on load. Not available on P/D fleets. +See [embeddings](REFERENCE.md#embeddings), [classify](REFERENCE.md#classify) +and [rerank](REFERENCE.md#rerank). -- `--tokenizer-backend fast` encodes with fastokens (decoding stays on HF). It - needs a `tokenizer.json` and falls back to `hf` when fastokens cannot load it. -- `--tokenizer-l1-cache-mb N` caches prefix tokenizations at special-token - boundaries, so a multi-turn chat encodes only the turns added since the - previous request. Boundaries are unconditional, non-normalized, non-stripping - special tokens with no overlapping added-token spellings. Unsafe candidates - are excluded; if none remain, encoding proceeds without the cache. +### Native gRPC -On a ~69K-token DeepSeek-V4 chat, `hf` encodes in ~40 ms, `fast` in ~4 ms, and -a new turn on a cached history in ~0.2 ms. The DeepSeek fixtures check every -case under `hf`, `fast`, and `fast` with L1. Startup logs report the resolved -backend and cache state. `/metrics` exposes only -`sgl_router_tokenizer_l1_tokens_total{source="cached"|"encoded"}` to measure -how much tokenization work the cache reuses. +`--grpc-port` serves the engine's `sglang.runtime.v1.SglangService`. Each RPC is +routed like its HTTP route and forwarded to the same RPC on the engine's own +native gRPC port, so engines must run with `--grpc-port` too. See +[native gRPC](REFERENCE.md#native-grpc). -## Native `/generate` +### Bucket file -`/generate` (`POST` or `PUT`) has the engine's interface: the same -`GenerateReqInput` body and the same response, buffered or SSE. The body names no -model, so requests go to the one this router serves. Worker selection -(`--chat-routing`, `--policy`), PD dispatch, and abort-on-disconnect are shared -with chat completions. +`--bucket-config` defines buckets whose groups set `worker_services` +(`namespace/name`) or `worker_ids`, plus `policy`, `admission` and `affinity`; +unset fields inherit the CLI flags. See +[ARCHITECTURE.md](ARCHITECTURE.md#2-configuration-examples) and +[bucket file membership](REFERENCE.md#bucket-file-membership). -With a tokenizer loaded, the router tokenizes `text` (a string or a list) with -the special tokens SGLang adds (BOS per `add_bos_token` for Llama-, Gemma- and -Cohere-class tokenizers, otherwise the `tokenizer.json` post-processor's), and -forwards it as `input_ids`. The engine skips tokenizing and routing sees its exact -tokens. Multimodal requests keep `text`, since the engine expands placeholders -from it, and `--disable-input-ids-forwarding` keeps it for every request. So does -a model whose `tokenizer.json` normalizer transformers replaces on load (legacy -SentencePiece Llama files, bge-m3), or whose `tokenizer_config.json` the router -cannot read, as the router cannot reproduce its tokens. A batch goes to one -worker: load counts every prompt and each of its `n` samples, while bucket and -context limits bound the longest prompt plus its own `max_new_tokens`. +### KV cache signals -Otherwise the body passes through, plus PD bootstrap fields, a minted `rid` for -a single prompt that has none, and `--override-sampling-params` defaults. Under -`--dp-aware` each worker's body also carries the chosen `routed_dp_rank`; a PD -batch or `n > 1` request leaves the prefill rank to the engine, which gives item -`i` the bootstrap room `room + i`. +- **External KV indexer**: `--cache-prefix-provider indexer + --kv-indexer-endpoint `; see [sgl-kv-indexer](sgl-kv-indexer/README.md) + and [external KV indexer](REFERENCE.md#external-kv-indexer). +- **KV-event gap replay**: add `"replay_endpoint"` to the engine's + `--kv-events-config` so the router can re-fetch dropped KV events; see + [gap replay](REFERENCE.md#kv-event-gap-replay). +- **Pending prefixes**: `--cache-pending-prefix-ttl-ms` keeps bursts that share + a cold prefix on one worker until its KV events arrive; see + [pending prefixes](REFERENCE.md#pending-prefixes). +- **Peer bootstrap (Kubernetes)**: `--kv-peer-selector + kubernetes.io/service-name=` makes a new replica copy the KV + tree from a sibling before `/readyz` passes; see + [peer bootstrap](REFERENCE.md#peer-bootstrap-kubernetes). +- **LoRA cache identity**: with workers that publish `BlockStored.lora_name`, + prefix matches are kept per adapter; see + [LoRA cache identity](REFERENCE.md#lora-cache-identity). -## OpenAI over /generate +### Operations -`/v1/completions` and `/v1/chat/completions` behave as the engine's own routes, -but the router runs SGLang's OpenAI layer itself (`sglang-processor`'s `openai` module): it lowers -the request to the `GenerateReqInput` SGLang would build, sends it to -`/generate` like a native request, and builds the OpenAI response, buffered or -SSE, from the engine's output. The engine's `/server_info` supplies the server -args that layer reads (`--enable-cache-report`, -`--stream-response-default-include-usage`, `--incremental-streaming-output`, the -custom-labels header). Requests it cannot reproduce exactly go to the engine's -own route as sent: workers that disagree on those args or never -reported them, `--completion-template`, `return_hidden_states` and other -`sglext` outputs, `echo` or `logprobs` without a router tokenizer, and bodies -SGLang would reject. Chat also needs a renderer verified against SGLang (the -`AllText` scope: DeepSeek-V4, V4.1, GLM-5.3, Kimi-K3 and Qwen3.8 today), router `--default-chat-template-kwargs` -equal to the workers', a reasoning parser the processor reproduces -(`deepseek-v4`, `deepseek-v41`, `glm45`, `kimi_k3`, `qwen3`) or none, and no media yet. Tools are served with `tool_choice` -`auto` or `none`, non-strict, with standard JSON-schema types, and a tool -parser the processor reproduces (`deepseekv4`, `deepseekv41`, `kimi_k3`, `qwen3_coder`); `required`, named and strict -tools take a constraint from the engine's xgrammar, so they go to the engine -(`kimi_k3`'s `auto` tag is reproduced). -With `--tool-call-parser glm47`, a chat with tools also goes to the engine: its -grammar needs `ebnf_full_assistant`, which `/generate` does not accept. -The router reads the model's `config.json` and `generation_config.json` beside -`--tokenizer-path` (or from the HF repo), as the engine reads them, so a local -path should point into the model's directory. Engine env vars that change the -layer (`SGLANG_TOOL_STRICT_LEVEL`, `SGLANG_FORWARD_UNKNOWN_TOOLS`) are taken as -unset. `sgl_router_openai_route_total` counts each outcome. - -## Embeddings - -`/v1/embeddings` has the engine's interface: the same OpenAI `EmbeddingRequest` -body and response. As for chat completions, `model` must name the served model. -A PD fleet answers 400, since prefill and decode engines serve no embeddings. - -Text `input` (a string or a list) is tokenized and forwarded as token IDs, as for -[`/generate`](#native-generate), plus the EOS SGLang appends for EmbeddingGemma. -Blank prompts and multimodal items stay as sent, for the engine to reject or -render. A list is a batch for one worker, routed on load. A single prompt with no -`rid` gets a minted one. Under `--dp-aware` the engine picks the rank, since its -embeddings endpoint reads none. - -## Classify - -`/v1/classify` has the engine's interface: the same `ClassifyRequest` body and -response. It is served as embeddings are, except that a batch of text stays text, -since the engine takes token IDs for only one prompt. - -## Completions - -`/v1/completions` has the engine's interface and is prepared like chat (sampling -contract, PD bootstrap, DP rank), with the prompt forwarded as sent: tokens the -router computes for routing do not replace it. - -## Rerank - -`/v1/rerank` has the engine's interface: the same `V1RerankReqInput` body and -response, sent as `POST` or `PUT` to the model this router serves. The body is -forwarded as sent, since the engine renders and tokenizes each query-document -pair. Routing is on load, with each pair's size estimated from its text. As for -embeddings, a PD fleet answers 400 and `--dp-aware` pins no rank. - -## Anthropic Messages - -`/v1/messages` converts each request to a chat completion at ingress, so it is -routed exactly like `/v1/chat/completions`, and converts the reply (or stream) -back to the Messages format. Consecutive turns of the same role are merged, as -the Messages API does, and a final assistant turn is sent as a prefill -(`continue_final_message`). Besides the Messages fields, only -`chat_template_kwargs` and `continue_final_message` reach the chat request. -Every error uses the Anthropic error envelope; 5xx messages are replaced with a -generic one, and a failed stream still closes its open block and ends with -`message_stop`. - -`/v1/messages/count_tokens` is answered by the router from the model's chat -template, without calling an engine. Image tokens are not counted, and a model -without a chat template answers 500 rather than a raw-text estimate. - -`thinking` sets the `thinking` / `enable_thinking` chat template kwargs. The -engine's own `/v1/messages` also adapts it per reasoning parser (for example -`reasoning_effort` for Hunyuan), which the router cannot see; on such models -use `output_config.effort` or `chat_template_kwargs` directly. - -## DeepSeek V4 and V4.1 - -Native V4 and V4.1 rendering comes from `rust/sglang-processor`, shared with SGLang's Rust -server, and follows SGLang's serving path (`serving_chat.py`), not Dynamo's -OpenAI defaults: all declared tools are rendered with SGLang's schema -defaults, reasoning effort comes from `reasoning` / `reasoning_effort`, and the -official/preview effort profile is detected from the checkpoint's -`encoding/encoding_dsv4.py` or overridden by `dsv4_reasoning_effort_profile` in -`config.json`, as in SGLang. Reference prompts live in `tests/fixtures/deepseek/` -and are regenerated by `tests/scripts/generate_deepseek_parity.py`. - -V4.1 Flash uses Dynamo's V4.1 encoder with SGLang's numeric reasoning budgets -and tool payloads. Developer messages and media are left to the worker, since -the encoder renders them differently. - -## GLM-5.3 - -GLM-5.3 and GLM-5.3-Flash render with the checkpoint's `chat_template.jinja` through -`rust/sglang-processor`, which builds the template context SGLang's Jinja path builds, -so every text chat forwards `input_ids`. Media, a `chat_template` kwarg, `-0` in tools -or tool arguments, and U+001C..U+001F in assistant history render on the worker. With -the cookbook's parsers, chats without tools are served through `/generate`; chats with -tools keep the engine's route (see above). - -## Qwen3.8 - -Qwen3.8 (Max, 27B and Flash-Next) renders with the checkpoint's `chat_template.jinja` -through `rust/sglang-processor`, which picks it by its template and builds the -template context SGLang's Jinja path builds. Every text chat forwards `input_ids`. -Media, a `chat_template` key in `chat_template_kwargs`, and text or tool numbers -Python's `trim` or `json.dumps` treat unlike Dynamo's (such as `1e-06`) render on the -worker. With the cookbook's `qwen3` and `qwen3_coder` parsers, text chats are served -through `/generate`; thinking follows the template's `enable_thinking` (on by default). - -## Kimi-K3 - -Kimi-K3 renders through `rust/sglang-processor` (dynamo-render's XTML formatter -with SGLang's request semantics) and is tokenized with the checkpoint's chunked -tiktoken encoding; shapes the formatter renders differently go to the worker. -`--tokenizer-path` accepts a -local `tiktoken.model` or an HF repo id, whose `tiktoken.model`, `config.json` -and `tokenizer_config.json` are downloaded when it has no `tokenizer.json`. -With the cookbook's `kimi_k3` parsers, text chats are served through `/generate`. - -## HTTP/2 - -There is nothing to configure. The router negotiates per connection inbound and -resolves the protocol per worker outbound; every combination below is reached -automatically, and HTTP/1.1 remains a supported peer on both edges. - -**Inbound.** The listener accepts cleartext HTTP/2 (h2c, prior knowledge) and -HTTP/1.1 on the same `--port`, chosen per connection. A mesh sidecar or -load balancer that prefers h2c multiplexes over one connection instead of -opening one per request; an HTTP/1.1 client is unaffected. - -**Outbound.** At registration the router reads each worker's `/server_info` and -forwards over cleartext h2c only when that worker reports `--enable-http2` (the -engine's Granian server, which serves h2c and HTTP/1.1 together) **and** the -worker URL is cleartext. Everything else uses the default client, which speaks -HTTP/1.1 in cleartext and negotiates ALPN `h2, http/1.1` over TLS: - -| `/server_info` | worker URL | router forwards over | -|---|---|---| -| `enable_http2: true` | `http://` | cleartext h2c | -| `enable_http2: true` | `https://` | HTTP/2 over TLS, by ALPN | -| `enable_http2: false`, or absent | any | HTTP/1.1 (cleartext) or ALPN (TLS) | - -The choice is per worker, not fleet-wide, so a mixed fleet works and one -worker's state never changes another's. The flag comes from the `/server_info` -launch record, which has reported it all along, so no engine change is needed. -A worker whose `/server_info` does not answer has no readable protocol and -forwards over HTTP/1.1 for as long as it stays registered — a throughput cost, -never a correctness one. Admin fan-out (`/flush_cache`) always uses the default -client, because it addresses every worker at once rather than a selected one. - -Two things worth knowing when debugging. h2c is prior-knowledge only — there is -no negotiation and no fallback — which is why the router requires the engine's -own `enable_http2` report before using it. And a worker's protocol is fixed -for as long as that worker stays registered: it is read once, at registration, -and never re-read. Under the K8s backend a restarting engine flips its -EndpointSlice to `ready=false`, which is a `Removed` → `Added` cycle and so a -fresh reading; under `--worker-urls` the fan-out happens once at startup and -nothing re-registers. So a worker registered over h2c that later stops serving -it (a proxy interposed on its port, say) is not detected until it -is re-registered; its circuit breaker will open in the meantime. - -## Native gRPC - -`--grpc-port` serves the engine's own gRPC API, `sglang.runtime.v1.SglangService` -(`proto/sglang/runtime/v1/sglang.proto`), so a gRPC client can point at the -router or at an engine. A call is routed exactly like its HTTP route and sent to -the same RPC on the selected engine's native gRPC server, which engines run -beside HTTP with their own `--grpc-port`; the router reads that port from each -worker's `/server_info`. HTTP requests are unaffected and still reach engines -over HTTP. - -| RPC | Routed like | -|---|---| -| `ChatComplete` | `/v1/chat/completions`: same rendering, `input_ids` forwarding, PD and DP-rank handling | -| `Complete` | `/v1/completions` | -| `Generate`, `TextGenerate` | `/generate` with `input_ids` or `text` | -| `Embed`, `TextEmbed`, `OpenAIEmbed` | `/v1/embeddings` | -| `Classify`, `OpenAIClassify` | `/v1/classify` | -| `Rerank` | `/v1/rerank` | - -The OpenAI RPCs carry the HTTP route's JSON body, so they are prepared exactly as -over HTTP. A typed RPC is prepared from its fields with the same rules, and the -router's additions (rid, PD bootstrap, DP rank, sampling defaults) are set on the -proto it forwards. A text prompt is tokenized once, as its HTTP route forwards -`input_ids`: `TextGenerate` is sent as `Generate` with `return_text`, so the engine -still supplies the text, and `TextEmbed` as `Embed`. They stay text RPCs under -`--disable-input-ids-forwarding`, without a tokenizer, or, for `TextGenerate`, with -`return_text_in_logprobs`. Other RPCs, such as -`Tokenize`, `HealthCheck` and the admin RPCs, answer `UNIMPLEMENTED`. Request metadata and `trace_headers` -act as HTTP headers for routing and forwarding, and a cancelled call aborts the -engine request. Router errors arrive as a gRPC status carrying the HTTP path's -message and an `x-router-error-code` trailer; an engine's status passes through -unchanged. An engine without `--grpc-port` (including one launched with -`--api-key`, which native gRPC rejects) cannot serve gRPC calls: the router warns -when it registers, and calls routed to it fail with `UNAVAILABLE`. - -## Upgrading from `cache_aware_zmq` - -The `cache_aware_zmq` policy has been removed. Configurations using it should -select `--policy cache_aware` and choose a native cache-prefix source: the -Router-local radix tree (the default), or the external Indexer shown above. - -The legacy `--cache-threshold`, `--balance-abs-threshold`, and -`--balance-rel-threshold` flags have also been removed. They do not have -one-to-one replacements; remove them and review the current `sgl-router ---help` output when tuning Cache-Aware routing. +- **No tokenizer**: `--no-tokenizer` for load-only policies (`power_of_two`, + `session_aware`); see [optional tokenizer](REFERENCE.md#optional-tokenizer-for-load-only-routing). +- **Retries**: `--retry-max-attempts N` resends a request that failed before + any response to another worker; see [retries](REFERENCE.md#retries). +- **DP-rank routing**: `--dp-aware` also picks the DP rank inside multi-rank + engines, and `--dp-rank-policy round_robin` cycles ranks that no routing key + or cached prefix decides; see [DP-rank routing](REFERENCE.md#dp-rank-routing). +- **PD version groups**: `--pd-version-group-label ` pairs prefill and + decode only within the same engine version; see + [PD version groups](REFERENCE.md#pd-version-groups). +- **Engine API key**: `--worker-api-key` for engines launched with `--api-key`; + see [engines with `--api-key`](REFERENCE.md#engines-with---api-key). +- **Sampling contract**: `--override-sampling-params ''` pins sampling + parameters fleet-wide; `--sampling-param-conflict reject|allow` decides what + happens to requests that differ. See + [fleet-wide sampling contract](REFERENCE.md#fleet-wide-sampling-contract) + for parameter domains and null handling. +- **Chat rendering**: use the same model files and chat-template settings on + router and workers, or set `--disable-input-ids-forwarding`. + `--tokenizer-backend fast` and `--tokenizer-l1-cache-mb N` speed up + tokenization. +- **HTTP/2**: no configuration. h2c is used inbound when the client speaks it, + and outbound to workers that report `enable_http2`; see + [HTTP/2](REFERENCE.md#http2). +- **Upgrading**: the `cache_aware_zmq` policy and the legacy cache and balance + threshold flags were removed; see + [upgrading](REFERENCE.md#upgrading-from-cache_aware_zmq). ## License diff --git a/experimental/sgl-router/REFERENCE.md b/experimental/sgl-router/REFERENCE.md new file mode 100644 index 0000000000..b4a5d50214 --- /dev/null +++ b/experimental/sgl-router/REFERENCE.md @@ -0,0 +1,640 @@ +# sgl-router reference + +Detailed behavior of the features summarized in [README.md](README.md). Routing +design, policies and bucket configuration are in [ARCHITECTURE.md](ARCHITECTURE.md). + +- [Discovery and KV signals](#discovery-and-kv-signals) +- [Bucket file membership](#bucket-file-membership) +- [Routing and reliability](#routing-and-reliability) +- [Chat rendering](#chat-rendering) +- [Native `/generate`](#native-generate) +- [OpenAI over /generate](#openai-over-generate) +- [Embeddings](#embeddings), [Classify](#classify), [Completions](#completions), [Rerank](#rerank) +- [Anthropic Messages](#anthropic-messages) +- Per-model rendering: [DeepSeek V4 and V4.1](#deepseek-v4-and-v41), [GLM-5.3](#glm-53), [Qwen3.8](#qwen38), [Kimi-K3](#kimi-k3) +- [HTTP/2](#http2) +- [Native gRPC](#native-grpc) +- [Upgrading from `cache_aware_zmq`](#upgrading-from-cache_aware_zmq) + +## Discovery and KV signals + +### PD version groups + +To run two engine versions side by side in PD mode (e.g. during a rollout), +pass `--pd-version-group-label `. A prefill worker is then paired only with +decode workers that have the same value for that EndpointSlice label (inherited +from the Service, so use one Service per role and version), so KV never crosses +versions. Unlabeled workers form their own group. + +```bash +sgl-router --model-id qwen3 --service-discovery \ + --prefill-selector app=engines-qwen3,role=prefill \ + --decode-selector app=engines-qwen3,role=decode \ + --pd-version-group-label sglang.ai/version-group +``` + +### External KV indexer + +```bash +sgl-router \ + --model-id qwen3 \ + --tokenizer-path /models/qwen3/tokenizer.json \ + --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \ + --policy cache_aware \ + --cache-prefix-provider indexer \ + --kv-indexer-endpoint http://10.0.0.10:50051 \ + --kv-indexer-query-timeout-ms 100 \ + --kv-indexer-query-max-inflight 32 +``` + +The Indexer replaces the Router-local radix tree as the native Cache-Aware +signal. Query timeouts and local concurrency are bounded by the two Indexer +options, which default to 100 ms and 32 respectively. + +### LoRA cache identity + +LoRA routing requires workers that publish `BlockStored.lora_name`. Requests +select the adapter through `model: "base:adapter"` on OpenAI routes, or +`lora_path` on native routes. Salted generation requests also carry `cache_salt`. + +Worker block IDs are opaque and may differ across adapter loads or workers. +The router derives adapter-aware lookup keys from event tokens and keeps a +per-worker, per-DP-rank mapping for parent links and removals. Multiple block +IDs can own the same lookup key; removing one leaves the other owners and +storage tiers intact. The remote indexer bridge performs the same translation. +Base-model events retain their existing published-key path. + +New replicas request `?include_lora=true` snapshots containing LoRA mappings +(version 2). Legacy consumers receive a base-only version-1 view, so mixed +snapshots do not prevent base-cache bootstrap during rolling upgrades. Base-only +snapshots retain version 1 for every consumer. LoRA stores buffered during bootstrap retain their +tokens; the original base-model buffering and replay behavior is unchanged. + +Reloading different adapter weights under the same public name can still +credit stale engine blocks until their eviction. Names identify adapters; +versioned names are needed to distinguish different weights. + +### KV-event gap replay + +ZMQ drops KV events at the publisher's high-water mark, and a lost removal +leaves the router crediting a worker with blocks it no longer holds. Give each +engine a replay socket and the router re-fetches the batches a sequence gap +skipped: + +```bash +python -m sglang.launch_server ... \ + --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558"}' +``` + +Rank r publishes on 5557 + r and replays on 5558 + r, so with `dp_size > 1` +place the replay base at least dp_size ports past the PUB base. The engine keeps +the last `buffer_steps` batches (default 10000); older gaps stay unrepaired. +`sgl_router_kv_event_replays_total{outcome}` counts the attempts. + +### Pending prefixes + +Until the engine's KV events arrive, requests that share a cold prefix see no +cached owner and spread across workers, each prefilling the same prefix. This +is common with parallel sampling, RL rollouts and agent fan-out. +`--cache-pending-prefix-ttl-ms` credits a worker with a prompt's prefix for that +long after routing it there, so the burst stays together. It is off by default +and needs `--policy cache_aware` with the Router-local radix tree. +`sgl_router_cache_pending_prefix_hits_total` counts lookups where a pending +prefix matched deeper than any confirmed one; the policy's candidate and +admission filters still decide whether that worker is picked. + +### Peer bootstrap (Kubernetes) + +A replica that starts mid-fleet subscribes to each worker's KV topic +mid-stream, so everything already resident in the engines' caches is invisible +to it and it routes cache-blind until traffic re-stores those blocks — +degrading the engines' locality for the whole fleet, once per replica on every +rolling update. With a peer selector set, a booting replica instead pulls a +tree snapshot from a warm sibling over `/internal/kv_snapshot` and splices it +under its own live delta stream, and `/readyz` stays 503 until that bootstrap +settles. Off unless `--kv-peer-selector` is set; requires `--policy +cache_aware` with the Router-local radix tree (not an external Indexer) and +`--service-discovery`, which the CLI enforces. + +```bash +sgl-router \ + --model-id qwen3 --policy cache_aware \ + --service-discovery --service-discovery-namespace prod \ + --selector app=engines-qwen3 \ + --kv-peer-selector kubernetes.io/service-name=sgl-router +``` + +- `--kv-peer-selector` is matched against **EndpointSlice labels** — the + router Service's labels plus `kubernetes.io/service-name`, never the pods' + labels, so `kubernetes.io/service-name=` is the unambiguous + choice. A pod-template label matches nothing and every replica boots cold. +- `--kv-bootstrap-timeout-ms` (default 600000) bounds the whole bootstrap; + readiness waits on it, so startup/readiness probes must tolerate a replica + staying unready this long. +- `--kv-bootstrap-fetch-timeout-cap-ms` (default 300000) caps one snapshot + fetch within that budget. +- `--kv-bootstrap-seed-required` (off by default) keeps holding `/readyz` when + siblings were present but their tree could not be pulled, bounded at + max(3× the bootstrap timeout, 60s) — it delays a failed seed's replica (and + with it a rolling update) rather than shipping it cache-blind; it cannot + stall a rollout indefinitely. + +The deployment needs two things beyond the flags: the router's ServiceAccount +must hold `get`/`list`/`watch` on `endpointslices` in the watched namespace +(the worker-discovery Role already grants this), and the pod spec should wire +`POD_NAME`, `POD_NAMESPACE` and `POD_IP` from the downward API so a replica +can exclude itself from its own peer list. A rolling update should surge +(`maxUnavailable: 0`) so new replicas always have warm siblings to copy from. +`tests/e2e/k8s_integration/manifests/kv-bootstrap.yaml` is a worked example of +all of it, and the `sgl_router_kv_bootstrap_*` series in +[monitoring/README.md](monitoring/README.md) make the whole path observable. + +## Bucket file membership + +`worker_services` matches Kubernetes Services by `namespace/name`, using the +`kubernetes.io/service-name` label on watched EndpointSlices. Replacement pods +and new replicas join the same group automatically. The router's discovery +selectors must include those EndpointSlices; this field does not expand the watch. +A worker selected by several Services belongs to each of them. + +For static URL discovery, use `"worker_ids": ["http://worker:30000"]`: each ID +is the configured worker URL. Kubernetes worker IDs are `namespace/pod-UID` +(and change when a pod is replaced), so use `worker_services` for durable pools. +Set only one membership field, or omit both for every engine of the group's role. +Empty membership lists and blank worker IDs are rejected at startup. + +Admission fields left unset or set to `null` inherit CLI defaults. To apply a +limit only to selected groups, omit that CLI default and set it on those groups. + +## Routing and reliability + +### Optional tokenizer for load-only routing + +`--no-tokenizer` skips tokenizer loading for load-only policies such as +`power_of_two` and `session_aware`, on either routing path. Workers tokenize the +original messages, and `/v1/tokenize` and `/v1/detokenize` are unavailable. +Cache-aware routing, prefix-cache terms or filters, and buckets with token-length +or context limits require a tokenizer. Reorg buckets that only select membership +and load-based policies can use `--no-tokenizer`. + +### DP-rank routing + +An engine launched with `--dp-size` or `--attn-dp-size` runs several DP ranks, +each with its own KV cache, behind one endpoint. With `--dp-aware`, the router +also picks the rank inside the selected worker. It sends that rank as +`X-Data-Parallel-Rank`, which the engine honors, and overwrites any value the +client sent. The router picks the first of these that applies: + +1. A hash of the sticky routing key or session id, so a conversation keeps + its rank and router replicas agree. +2. The rank with the deepest cached prefix in the local KV tree. +3. The rank with the fewest requests this router has in flight on it, or with + `--dp-rank-policy round_robin`, the worker's ranks in turn. + +In PD mode, decode is ranked by step 3 only. The decode request names the +prefill rank in `disagg_prefill_dp_rank`, and the bootstrap room satisfies +`room % prefill_dp_size == prefill_rank`. + +### Retries + +`--retry-max-attempts N` (default 1, which disables retries; 3 is typical) lets +a request that fails before any response reaches the client be sent again to a +worker it has not tried yet. A failure is a transport error, an open circuit +breaker, a 5xx, a 429, or a timeout: `--request-timeout-secs` bounds a whole +non-streaming response, and a streaming one until its headers arrive. SGLang's +chat endpoint sends those headers with the first token, so for streaming chat +the same flag bounds time to first token, queueing included; raise it for +long-context or deeply queued fleets. Once a streaming response's 2xx status +has been sent, it is never retried, and neither is any other 2xx or 4xx. In PD mode the failed +side is excluded, or both sides when the caller set `rid`, which an engine still +running it would refuse; a new pair gets a new bootstrap room. Retries share the +request's `--stale-request-timeout-secs` deadline, and none starts after it. When +every eligible worker has failed, the client gets the last failure. +`sgl_router_retries_total` counts the retried attempts. + +Each retry first waits out a jittered exponential backoff: a random delay in +`[d/2, d]`, where `d` starts at `--retry-initial-backoff-ms` (default 50), +doubles per retry, and is capped at `--retry-max-backoff-ms` (default 2000). +Set the initial backoff to 0 to retry immediately. A backoff never waits past +the request's stale deadline. + +### Engines with `--api-key` + +The router reads each worker's `/server_info` and `/model_info` to learn its +model, KV-event publisher, HTTP/2 support and DP size. An engine launched with +`--api-key` rejects those requests without the key, so pass the same key as +`--worker-api-key`. Chat requests and `/flush_cache` forward the caller's +`Authorization` header instead, so callers still need the engine key. + +### Fleet-wide sampling contract + +`--override-sampling-params` fixes the sampling configuration for every client +of this router, independently of what the engine's own defaults happen to be +(on native [`/generate`](#native-generate), in each prompt's `sampling_params`): + +```bash +sgl-router \ + --model-id qwen3 \ + --tokenizer-path /models/qwen3/tokenizer.json \ + --worker-urls http://10.0.0.1:30000 \ + --override-sampling-params '{"temperature": 1, "top_p": 0.95, "n": 1}' \ + --sampling-param-conflict reject +``` + +It takes one JSON object keyed by the request-body field names +(`temperature`, `top_p`, `top_k`, `min_p`, `repetition_penalty`, +`frequency_penalty`, `presence_penalty`, `n`). A configured value is injected +whenever the request omits that field, so the engine's defaults cannot drift +from what the operator declared. `temperature`, `top_p`, `top_k`, `min_p` and +`repetition_penalty` are the five the engine resolves from the model's own +`generation_config`, which is what makes them drift when a deployed image +changes; the rest have fixed API defaults. + +An explicit `null` counts as omitting the field, not as a client-supplied +value: the OpenAI API types these parameters as nullable with a documented +default, so `null` asks for the default — and on a governed fleet the +configured value is what the default is. + +For a request that does send a value, `--sampling-param-conflict` decides: +`reject` (the default) 400s a differing value before admission, while `allow` +forwards the client's value untouched — the router never silently rewrites what +a client sent. A `reject` response carries +`x-router-error-code: sampling_contract_violation` and is counted in +`sgl_router_sampling_contract_rejections_total{param}`, so a rollout's blast +radius is visible per parameter rather than folded into `bad_request`. + +`reject` also 400s a value it cannot read as a number, on a governed parameter +only. The engine coerces more than JSON numbers — a bool and a numeric string +both become numbers — by rules that are undocumented and need not match across +a fleet, so a value the router cannot read is one it cannot prove conforms, and +waving it through would make the pin bypassable. The common coercions are +matched exactly (`false` is 0, `"0.5"` and `"1_0"` are 0.5 and 10), so this +refuses only genuine garbage. `allow` is unaffected: it promises nothing, so +such a value keeps flowing to the engine, which owns the request schema. + +A value may also be an inclusive band, `{"min": LO, "max": HI}`, for a +parameter that stays tunable inside a range. A band names no value to inject, +so it constrains only the requests that name the parameter; one that omits it +gets the model's own `generation_config` default, which the router cannot see. +Because a band can only ever reject, combining one with `allow` is a startup +error. + +Values are range-checked at startup, so a misconfiguration fails the launch +instead of 400ing every request at the engine. Repeating a key in the flag is +also a startup error, rather than silently enforcing whichever copy came last. + +| parameter | accepted | notes | +| --- | --- | --- | +| `temperature` | `[0, 2]` | | +| `top_p` | `(0, 1]` | | +| `top_k` | `>= 1`, or exactly `-1` | `-1` is the engine's "whole vocabulary" spelling and its default. Being non-contiguous it cannot bound a band. Note `top_k: 1` is greedy decoding, not "disabled". | +| `min_p` | `[0, 1]` | not an OpenAI parameter; the engine's domain | +| `repetition_penalty` | `(0, 2]` | not an OpenAI parameter; the engine's domain | +| `frequency_penalty` | `[-2, 2]` | | +| `presence_penalty` | `[-2, 2]` | | +| `n` | `[1, 128]` | | + +The OpenAI domains are deliberately narrower than what the engine itself +accepts (`SamplingParams.verify` would take `temperature: 5`): these values are +injected into request bodies, and a fleet contract outside the range every +OpenAI client library validates against is far more likely a typo than an +intent. + +Cost: a request that named every configured field forwards its original bytes +untouched. One that omits a field has the scalars spliced directly into the +body bytes — no parse, no re-serialize — which on a 16 MiB body is ~0.24 ms +against ~5.7 ms for a `serde_json` round-trip. Only `input_ids` and PD +bootstrap injection still parse and re-serialize, because those may have to +overwrite a key the client sent. + +### Relationship to the engine's own `--preferred-sampling-params` + +The engine has an inject-when-absent flag of its own, +`--preferred-sampling-params`, merged in +`python/sglang/srt/managers/tokenizer_manager.py` as +`{**preferred, **obj.sampling_params}`. On `/v1/chat/completions` it is +currently a no-op: `ChatCompletionRequest.to_sampling_params` +(`python/sglang/srt/entrypoints/openai/protocol.py`) resolves every sampling +key eagerly through `generation_config` and then its own defaults, so the +right-hand side of that merge is always fully populated and always wins. +(`/v1/responses`, in the same file, already omits `None` entries for exactly +this reason.) If that is fixed engine-side, `--preferred-sampling-params` +covers the inject-when-absent half for a single engine. + +What stays the router's to own either way is the enforcement half: the +`reject` immutability contract with its 400 before admission — costing no +queue slot and no engine round-trip — the `sampling_contract_violation` code +and per-parameter counter, bands, and one contract applied at a shared ingress +across engines whose own flags the router operator may not control. + +## Chat rendering + +The router renders chat requests with dynamo-render (`dynamo-renderer`): the model's +HF Jinja template from `tokenizer_config.json` or a sibling +`chat_template.jinja`, or dynamo-render's built-in DeepSeek encoder (V4 family, V3.2) +for template-less models. Cache-aware routing hashes the rendered tokens so its +prefix queries match the blocks the engine caches. Models the engine encodes in +code but dynamo-render cannot tokenize here (Inkling) route via raw prompt +text, as does any model whose template fails to load or render. + +Some chats additionally forward the rendered tokens to the engine as +`input_ids`, retaining the original messages, so the engine skips +re-tokenizing. How many depends on the model's renderer. DeepSeek-V4's, V4.1's +and Kimi-K3's native encoders and GLM-5.3's and Qwen3.8's templates are +fixture-verified against SGLang's request normalization, so they forward every +chat except multimodal ones and those with caller-provided `input_ids`. Renderers +without that verification (HF Jinja templates) +forward only plain text chat requests (string `content`, no tools, no template +kwargs or reasoning controls or historical `reasoning_content`, no assistant +continuation, no consecutive users or non-leading system turns) and warn +`UNVERIFIED` at startup; every other request shape is rendered for routing +only. + +Use matching model files on the router and workers, and set +the same `--default-chat-template-kwargs`, `SGLANG_DEFAULT_THINKING`, +`SGLANG_DSV4_REASONING_EFFORT`, and `SGLANG_DSV41_REASONING_EFFORT` on both. +The router reads these render defaults from its own configuration and environment; +it does not discover the workers' settings. Point `--tokenizer-path` at the workers' +model snapshot so the V4 effort profile is read from the same +`encoding/encoding_dsv4.py`. + +Set `--disable-input-ids-forwarding` for this router's model when worker-side +rendering has not been verified to match. This disables router-generated IDs +for every routing policy; cache-aware routing still renders and tokenizes +locally, and the original messages reach the workers for engine processing. +Caller-supplied `input_ids` remain caller-owned and pass through unchanged. + +Forwarding logs its assumptions at startup. In particular, disable it when the +router's render defaults differ from the workers', for worker template overrides +not reflected in the router's model files, for worker parser overrides such as +`--tool-call-parser deepseekv32` that select a native encoder over a shipped +template, or conversation templates with stop strings (the engine's `input_ids` +path skips those template stops). Disabling forwarding preserves engine behavior +but does not establish parity for local routing hashes. + +Also set `--disable-input-ids-forwarding` for array-only templates: Dynamo may wrap +string content into arrays differently from the worker. The pinned Dynamo renderer does not expose +its conversion flag, so the router cannot automatically block these templates. +Detailed content-format parity coverage follows in #39133. + +The Dynamo crates are pinned exactly and `Cargo.lock` is committed; CI builds +with `--locked`, so rendered bytes cannot change without a reviewed diff. + +Router tokenization sits on the TTFT path for every chat it renders. Two opt-in +flags make it cheaper: + +- `--tokenizer-backend fast` encodes with fastokens (decoding stays on HF). It + needs a `tokenizer.json` and falls back to `hf` when fastokens cannot load it. +- `--tokenizer-l1-cache-mb N` caches prefix tokenizations at special-token + boundaries, so a multi-turn chat encodes only the turns added since the + previous request. Boundaries are unconditional, non-normalized, non-stripping + special tokens with no overlapping added-token spellings. Unsafe candidates + are excluded; if none remain, encoding proceeds without the cache. + +On a ~69K-token DeepSeek-V4 chat, `hf` encodes in ~40 ms, `fast` in ~4 ms, and +a new turn on a cached history in ~0.2 ms. The DeepSeek fixtures check every +case under `hf`, `fast`, and `fast` with L1. Startup logs report the resolved +backend and cache state. `/metrics` exposes only +`sgl_router_tokenizer_l1_tokens_total{source="cached"|"encoded"}` to measure +how much tokenization work the cache reuses. + +## Native `/generate` + +`/generate` (`POST` or `PUT`) has the engine's interface: the same +`GenerateReqInput` body and the same response, buffered or SSE. The body names no +model, so requests go to the one this router serves. Worker selection +(`--chat-routing`, `--policy`), PD dispatch, and abort-on-disconnect are shared +with chat completions. + +With a tokenizer loaded, the router tokenizes `text` (a string or a list) with +the special tokens SGLang adds (BOS per `add_bos_token` for Llama-, Gemma- and +Cohere-class tokenizers, otherwise the `tokenizer.json` post-processor's), and +forwards it as `input_ids`. The engine skips tokenizing and routing sees its exact +tokens. Multimodal requests keep `text`, since the engine expands placeholders +from it, and `--disable-input-ids-forwarding` keeps it for every request. So does +a model whose `tokenizer.json` normalizer transformers replaces on load (legacy +SentencePiece Llama files, bge-m3), or whose `tokenizer_config.json` the router +cannot read, as the router cannot reproduce its tokens. A batch goes to one +worker: load counts every prompt and each of its `n` samples, while bucket and +context limits bound the longest prompt plus its own `max_new_tokens`. + +Otherwise the body passes through, plus PD bootstrap fields, a minted `rid` for +a single prompt that has none, and `--override-sampling-params` defaults. Under +`--dp-aware` each worker's body also carries the chosen `routed_dp_rank`; a PD +batch or `n > 1` request leaves the prefill rank to the engine, which gives item +`i` the bootstrap room `room + i`. + +## OpenAI over /generate + +`/v1/completions` and `/v1/chat/completions` behave as the engine's own routes, +but the router runs SGLang's OpenAI layer itself (`sglang-processor`'s `openai` module): it lowers +the request to the `GenerateReqInput` SGLang would build, sends it to +`/generate` like a native request, and builds the OpenAI response, buffered or +SSE, from the engine's output. The engine's `/server_info` supplies the server +args that layer reads (`--enable-cache-report`, +`--stream-response-default-include-usage`, `--incremental-streaming-output`, the +custom-labels header). Requests it cannot reproduce exactly go to the engine's +own route as sent: workers that disagree on those args or never +reported them, `--completion-template`, `return_hidden_states` and other +`sglext` outputs, `echo` or `logprobs` without a router tokenizer, and bodies +SGLang would reject. Chat also needs a renderer verified against SGLang (the +`AllText` scope: DeepSeek-V4, V4.1, GLM-5.3, Kimi-K3 and Qwen3.8 today), router `--default-chat-template-kwargs` +equal to the workers', a reasoning parser the processor reproduces +(`deepseek-v4`, `deepseek-v41`, `glm45`, `kimi_k3`, `qwen3`) or none, and no media yet. Tools are served with `tool_choice` +`auto` or `none`, non-strict, with standard JSON-schema types, and a tool +parser the processor reproduces (`deepseekv4`, `deepseekv41`, `kimi_k3`, `qwen3_coder`); `required`, named and strict +tools take a constraint from the engine's xgrammar, so they go to the engine +(`kimi_k3`'s `auto` tag is reproduced). +With `--tool-call-parser glm47`, a chat with tools also goes to the engine: its +grammar needs `ebnf_full_assistant`, which `/generate` does not accept. +The router reads the model's `config.json` and `generation_config.json` beside +`--tokenizer-path` (or from the HF repo), as the engine reads them, so a local +path should point into the model's directory. Engine env vars that change the +layer (`SGLANG_TOOL_STRICT_LEVEL`, `SGLANG_FORWARD_UNKNOWN_TOOLS`) are taken as +unset. `sgl_router_openai_route_total` counts each outcome. + +## Embeddings + +`/v1/embeddings` has the engine's interface: the same OpenAI `EmbeddingRequest` +body and response. As for chat completions, `model` must name the served model. +A PD fleet answers 400, since prefill and decode engines serve no embeddings. + +Text `input` (a string or a list) is tokenized and forwarded as token IDs, as for +[`/generate`](#native-generate), plus the EOS SGLang appends for EmbeddingGemma. +Blank prompts and multimodal items stay as sent, for the engine to reject or +render. A list is a batch for one worker, routed on load. A single prompt with no +`rid` gets a minted one. Under `--dp-aware` the engine picks the rank, since its +embeddings endpoint reads none. + +## Classify + +`/v1/classify` has the engine's interface: the same `ClassifyRequest` body and +response. It is served as embeddings are, except that a batch of text stays text, +since the engine takes token IDs for only one prompt. + +## Completions + +`/v1/completions` has the engine's interface and is prepared like chat (sampling +contract, PD bootstrap, DP rank), with the prompt forwarded as sent: tokens the +router computes for routing do not replace it. + +## Rerank + +`/v1/rerank` has the engine's interface: the same `V1RerankReqInput` body and +response, sent as `POST` or `PUT` to the model this router serves. The body is +forwarded as sent, since the engine renders and tokenizes each query-document +pair. Routing is on load, with each pair's size estimated from its text. As for +embeddings, a PD fleet answers 400 and `--dp-aware` pins no rank. + +## Anthropic Messages + +`/v1/messages` converts each request to a chat completion at ingress, so it is +routed exactly like `/v1/chat/completions`, and converts the reply (or stream) +back to the Messages format. Consecutive turns of the same role are merged, as +the Messages API does, and a final assistant turn is sent as a prefill +(`continue_final_message`). Besides the Messages fields, only +`chat_template_kwargs` and `continue_final_message` reach the chat request. +Every error uses the Anthropic error envelope; 5xx messages are replaced with a +generic one, and a failed stream still closes its open block and ends with +`message_stop`. + +`/v1/messages/count_tokens` is answered by the router from the model's chat +template, without calling an engine. Image tokens are not counted, and a model +without a chat template answers 500 rather than a raw-text estimate. + +`thinking` sets the `thinking` / `enable_thinking` chat template kwargs. The +engine's own `/v1/messages` also adapts it per reasoning parser (for example +`reasoning_effort` for Hunyuan), which the router cannot see; on such models +use `output_config.effort` or `chat_template_kwargs` directly. + +## DeepSeek V4 and V4.1 + +Native V4 and V4.1 rendering comes from `rust/sglang-processor`, shared with SGLang's Rust +server, and follows SGLang's serving path (`serving_chat.py`), not Dynamo's +OpenAI defaults: all declared tools are rendered with SGLang's schema +defaults, reasoning effort comes from `reasoning` / `reasoning_effort`, and the +official/preview effort profile is detected from the checkpoint's +`encoding/encoding_dsv4.py` or overridden by `dsv4_reasoning_effort_profile` in +`config.json`, as in SGLang. Reference prompts live in `tests/fixtures/deepseek/` +and are regenerated by `tests/scripts/generate_deepseek_parity.py`. + +V4.1 Flash uses Dynamo's V4.1 encoder with SGLang's numeric reasoning budgets +and tool payloads. Developer messages and media are left to the worker, since +the encoder renders them differently. + +## GLM-5.3 + +GLM-5.3 and GLM-5.3-Flash render with the checkpoint's `chat_template.jinja` through +`rust/sglang-processor`, which builds the template context SGLang's Jinja path builds, +so every text chat forwards `input_ids`. Media, a `chat_template` kwarg, `-0` in tools +or tool arguments, and U+001C..U+001F in assistant history render on the worker. With +the cookbook's parsers, chats without tools are served through `/generate`; chats with +tools keep the engine's route (see above). + +## Qwen3.8 + +Qwen3.8 (Max, 27B and Flash-Next) renders with the checkpoint's `chat_template.jinja` +through `rust/sglang-processor`, which picks it by its template and builds the +template context SGLang's Jinja path builds. Every text chat forwards `input_ids`. +Media, a `chat_template` key in `chat_template_kwargs`, and text or tool numbers +Python's `trim` or `json.dumps` treat unlike Dynamo's (such as `1e-06`) render on the +worker. With the cookbook's `qwen3` and `qwen3_coder` parsers, text chats are served +through `/generate`; thinking follows the template's `enable_thinking` (on by default). + +## Kimi-K3 + +Kimi-K3 renders through `rust/sglang-processor` (dynamo-render's XTML formatter +with SGLang's request semantics) and is tokenized with the checkpoint's chunked +tiktoken encoding; shapes the formatter renders differently go to the worker. +`--tokenizer-path` accepts a +local `tiktoken.model` or an HF repo id, whose `tiktoken.model`, `config.json` +and `tokenizer_config.json` are downloaded when it has no `tokenizer.json`. +With the cookbook's `kimi_k3` parsers, text chats are served through `/generate`. + +## HTTP/2 + +There is nothing to configure. The router negotiates per connection inbound and +resolves the protocol per worker outbound; every combination below is reached +automatically, and HTTP/1.1 remains a supported peer on both edges. + +**Inbound.** The listener accepts cleartext HTTP/2 (h2c, prior knowledge) and +HTTP/1.1 on the same `--port`, chosen per connection. A mesh sidecar or +load balancer that prefers h2c multiplexes over one connection instead of +opening one per request; an HTTP/1.1 client is unaffected. + +**Outbound.** At registration the router reads each worker's `/server_info` and +forwards over cleartext h2c only when that worker reports `--enable-http2` (the +engine's Granian server, which serves h2c and HTTP/1.1 together) **and** the +worker URL is cleartext. Everything else uses the default client, which speaks +HTTP/1.1 in cleartext and negotiates ALPN `h2, http/1.1` over TLS: + +| `/server_info` | worker URL | router forwards over | +|---|---|---| +| `enable_http2: true` | `http://` | cleartext h2c | +| `enable_http2: true` | `https://` | HTTP/2 over TLS, by ALPN | +| `enable_http2: false`, or absent | any | HTTP/1.1 (cleartext) or ALPN (TLS) | + +The choice is per worker, not fleet-wide, so a mixed fleet works and one +worker's state never changes another's. The flag comes from the `/server_info` +launch record, which has reported it all along, so no engine change is needed. +A worker whose `/server_info` does not answer has no readable protocol and +forwards over HTTP/1.1 for as long as it stays registered — a throughput cost, +never a correctness one. Admin fan-out (`/flush_cache`) always uses the default +client, because it addresses every worker at once rather than a selected one. + +Two things worth knowing when debugging. h2c is prior-knowledge only — there is +no negotiation and no fallback — which is why the router requires the engine's +own `enable_http2` report before using it. And a worker's protocol is fixed +for as long as that worker stays registered: it is read once, at registration, +and never re-read. Under the K8s backend a restarting engine flips its +EndpointSlice to `ready=false`, which is a `Removed` → `Added` cycle and so a +fresh reading; under `--worker-urls` the fan-out happens once at startup and +nothing re-registers. So a worker registered over h2c that later stops serving +it (a proxy interposed on its port, say) is not detected until it +is re-registered; its circuit breaker will open in the meantime. + +## Native gRPC + +`--grpc-port` serves the engine's own gRPC API, `sglang.runtime.v1.SglangService` +(`proto/sglang/runtime/v1/sglang.proto`), so a gRPC client can point at the +router or at an engine. A call is routed exactly like its HTTP route and sent to +the same RPC on the selected engine's native gRPC server, which engines run +beside HTTP with their own `--grpc-port`; the router reads that port from each +worker's `/server_info`. HTTP requests are unaffected and still reach engines +over HTTP. + +| RPC | Routed like | +|---|---| +| `ChatComplete` | `/v1/chat/completions`: same rendering, `input_ids` forwarding, PD and DP-rank handling | +| `Complete` | `/v1/completions` | +| `Generate`, `TextGenerate` | `/generate` with `input_ids` or `text` | +| `Embed`, `TextEmbed`, `OpenAIEmbed` | `/v1/embeddings` | +| `Classify`, `OpenAIClassify` | `/v1/classify` | +| `Rerank` | `/v1/rerank` | + +The OpenAI RPCs carry the HTTP route's JSON body, so they are prepared exactly as +over HTTP. A typed RPC is prepared from its fields with the same rules, and the +router's additions (rid, PD bootstrap, DP rank, sampling defaults) are set on the +proto it forwards. A text prompt is tokenized once, as its HTTP route forwards +`input_ids`: `TextGenerate` is sent as `Generate` with `return_text`, so the engine +still supplies the text, and `TextEmbed` as `Embed`. They stay text RPCs under +`--disable-input-ids-forwarding`, without a tokenizer, or, for `TextGenerate`, with +`return_text_in_logprobs`. Other RPCs, such as +`Tokenize`, `HealthCheck` and the admin RPCs, answer `UNIMPLEMENTED`. Request metadata and `trace_headers` +act as HTTP headers for routing and forwarding, and a cancelled call aborts the +engine request. Router errors arrive as a gRPC status carrying the HTTP path's +message and an `x-router-error-code` trailer; an engine's status passes through +unchanged. An engine without `--grpc-port` (including one launched with +`--api-key`, which native gRPC rejects) cannot serve gRPC calls: the router warns +when it registers, and calls routed to it fail with `UNAVAILABLE`. + +## Upgrading from `cache_aware_zmq` + +The `cache_aware_zmq` policy has been removed. Configurations using it should +select `--policy cache_aware` and choose a native cache-prefix source: the +Router-local radix tree (the default), or the external Indexer shown above. + +The legacy `--cache-threshold`, `--balance-abs-threshold`, and +`--balance-rel-threshold` flags have also been removed. They do not have +one-to-one replacements; remove them and review the current `sgl-router +--help` output when tuning Cache-Aware routing. diff --git a/experimental/sgl-router/assets/architecture-deployment.png b/experimental/sgl-router/assets/architecture-deployment.png new file mode 100644 index 0000000000..d5f7d0b01b Binary files /dev/null and b/experimental/sgl-router/assets/architecture-deployment.png differ diff --git a/experimental/sgl-router/src/config/cli.rs b/experimental/sgl-router/src/config/cli.rs index 43ee432b38..0bcd8cf92f 100644 --- a/experimental/sgl-router/src/config/cli.rs +++ b/experimental/sgl-router/src/config/cli.rs @@ -100,7 +100,7 @@ pub struct ModelArgs { /// Accepts temperature, top_p, top_k, min_p, repetition_penalty, /// frequency_penalty, presence_penalty, and n. Numeric values fill absent /// fields; {"min": LO, "max": HI} bands only constrain supplied values - /// and require reject mode. See README for domains and null handling. + /// and require reject mode. See REFERENCE.md for domains and null handling. #[arg(long, value_name = "JSON")] pub override_sampling_params: Option, diff --git a/experimental/sgl-router/src/policies_reorg/mod.rs b/experimental/sgl-router/src/policies_reorg/mod.rs index b0873c14d1..7cb23a88d4 100644 --- a/experimental/sgl-router/src/policies_reorg/mod.rs +++ b/experimental/sgl-router/src/policies_reorg/mod.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 The SGLang Authors // SPDX-License-Identifier: Apache-2.0 -//! Side-by-side implementation of POLICY_DESIGN.md. Chat routing can opt into +//! Routing policies described in ARCHITECTURE.md. Chat routing can opt into //! this interface through AppContext; `policies` remains the default. pub mod admission;