[Doc] sgl-router: Rewrite README and add architecture guide (#43592)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
This commit is contained in:
Kan Wu
2026-10-18 13:29:08 +08:00
committed by GitHub
co-authored by Claude Opus 5.5 Shangming Cai
parent e5dc10e73b
commit 62d4261cf4
7 changed files with 1167 additions and 1331 deletions
+236
View File
@@ -0,0 +1,236 @@
# SGLang router architecture
This document describes `--chat-routing reorg`: the deployment, request flow,
and routing policies. The default legacy path is outside its scope.
- [1. Overview](#1-overview)
- [2. Configuration examples](#2-configuration-examples)
## 1. Overview
### 1.1 Deployment Architecture
A deployment consists of a **router** and **buckets of SGLang engines**. A bucket
is a logical group defined in the router configuration. It contains either a
**unified engine group**, where each engine handles both prefill and decode,
or separate **prefill and decode (P/D) groups**.
The example below shows short-context and long-context buckets, each with prefill
and decode engines.
![Router deployment with short-context and long-context P/D engine buckets](./assets/architecture-deployment.png)
Each engine publishes load reports and KV events, which the router uses for
routing. An optional [external KV indexer](sgl-kv-indexer/README.md) can supply
prefix matches. The router also maintains session bindings and in-flight request
counts.
To serve a request, the router chooses a bucket, selects engines within it, and
forwards the request. In P/D, it sends the request to both the prefill and decode
engines; the prefill engine transfers KV directly to the decode engine.
### 1.2 Request routing lifecycle
Routing a request typically goes through **preprocessing** (rendering and
tokenizing), **picking a bucket**, **scoring candidate engines** (load, prefix
matching, or session affinity), **choosing engine(s)**, **forwarding** the
request, and **handling output tokens** (e.g. tool parsing). Along the way it
also applies **admission control**, **failure handling**, and **retries**.
### 1.3 Example: load-aware routing
The router tries each **bucket** that fits the request's length, ordered by SLO
preference, capacity, then `rank`; a request that fits no bucket is rejected
with **400** before any engine is considered. In each bucket it drops the
engines that fail **admission**, then picks the **less loaded of two engines
sampled** from those that remain; when no engine admits the request it moves to
the next bucket, and returns **503** when no bucket has one. If **dispatch**
fails before any response and **retries** remain, it excludes that engine and
restarts from the first bucket.
**Load** is a vector of engine metrics compared field by field, and the first
difference decides. For **unified and prefill** engines the order is estimated
prefill queue time, waiting uncached tokens, waiting requests, running requests,
then in-flight requests; **decode** engines rank by waiting and running requests,
then KV usage.
```mermaid
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#eaf3fb", "primaryTextColor": "#172b4d", "primaryBorderColor": "#345779", "lineColor": "#345779", "edgeLabelBackground": "#ffffff"}, "flowchart": {"curve": "stepAfter", "rankSpacing": 30, "nodeSpacing": 30}}}%%
flowchart TD
R([Request]) --> L{"0. Fits any bucket's<br/>length limits?"}
L -->|no| Y(["400 to client"])
L -->|yes| B["1. Take next compatible bucket"]
B --> A{"2. Any engine can admit<br/>the request?"}
A -->|yes| S["3. Sample two admitted engines<br/>Pick the less loaded one"]
S --> D["4. Dispatch to engine"]
A -->|no| M{"More buckets?"}
M -->|no| X(["503 to client"])
D -->|success, or<br/>out of retries| E([Response to client])
D -->|failed before response,<br/>retries left| T["Exclude failed engine"]
M -.->|yes: next bucket| B
T -.->|restart at first bucket| B
classDef next fill:#fff5e6,stroke:#ac762c,color:#654516;
classDef done fill:#e8f5ef,stroke:#46826a,color:#234d3c;
class T,X,Y next;
class E done;
```
### 1.4 Example: cache-aware and session-aware routing
These policies replace steps **2–3** above. `cache_aware` serves plain and
prefill groups only; decode groups use load-aware (`power_of_two`) unless they set
`session_aware`. Both policies split the group's engines in two:
- **Preferred engine (A):** for `cache_aware`, the best-ranked prefix owner that
passes admission; for `session_aware`, the engine bound to the session.
- **Normal engines:** the rest of the engine group. When needed, the router picks
one of them (**B**) with the load-aware `power_of_two` policy.
A and B are checked against the same group admission limits (for example
`max_kv_usage` or `max_waiting_requests`; see [configuration examples](#2-configuration-examples)).
- **`prefer` mode (default):** use A if it passes admission.
- **`balanced` mode:** also pick a B engine and compare the loads of A and B.
- Switch to B only when it is clearly less loaded:
`load(A) > load_factor × load(B)` **and** `load(A) − load(B) > load_gap`
(defaults: factor 2; gap 1024 tokens, or 4 requests). Otherwise, including
when either load is unknown, keep A.
- `balanced_by` sets the load metric:
- **`prefill_tokens` (default):** the engine's reported waiting uncached tokens,
plus the tokens this request would prefill on it. For `cache_aware` that
subtracts the engine's cached prefix, so A's cache hit counts in its favor.
- **`running_requests`:** the engine's reported running requests.
- **No A, or A fails admission (either mode):** fall back to B.
## 2. Configuration examples
**CLI flags** set the defaults for every engine group. A **bucket file**
(`--bucket-config`) defines the buckets, and each group in it can override the
policy, admission, and affinity settings for itself. Without a bucket file,
there are no length tiers: every request can go to any engine.
Settings fall into these categories:
| Category | What it controls | CLI flags | Bucket-file group key |
| --- | --- | --- | --- |
| **Engines** | Plain or P/D, and where engines come from | `--worker-urls`, or `--service-discovery` with `--selector` (plain) or `--prefill-selector` + `--decode-selector` (P/D) | `worker_services`, `worker_ids` |
| **Policy** | How the router scores engines within a group | `--policy` | `policy` |
| **Affinity** | When the preferred engine (prefix owner or session binding) wins over a normal engine | `--affinity-*`, `--cache-affinity-*`, `--session-id-header` | `affinity` |
| **Admission** | Limits that make an engine reject a request | `--max-*` | `admission` |
| **Buckets** | Which buckets a request fits and the order they are tried | `--bucket-config` | `max_input_tokens`, `max_context_tokens`, `rank`, ... |
How policies apply to each group:
- **Plain and prefill groups** use `--policy` (default `power_of_two`).
- **Decode groups** always default to `power_of_two`; under reorg,
`--decode-policy` must stay `power_of_two`.
- **A bucket-file group** can set `"policy"` to `power_of_two` or to the
`--policy` value, nothing else. `cache_aware` cannot be used on decode, so
only `session_aware` can replace `power_of_two` there.
### Example A: plain engines, cache-aware
```bash
sgl-router --model-id qwen3 --chat-routing reorg \
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
--policy cache_aware \
--affinity-mode prefer --cache-affinity-min-match-ratio 0.2 \
--max-waiting-requests 16 --max-in-flight 56
```
- **Engines:** two plain engines; each handles both prefill and decode.
- **Policy:** every engine is scored with `cache_aware`.
- **Affinity:**
- An engine owns the prefix if it holds at least 1024 matched tokens (the
default) **and** 20% of the prompt.
- `prefer` sends the request to the best owner that passes admission.
- **Admission:** an engine is rejected when it reports 16 or more waiting
requests, or has 56 or more requests in flight from this router.
### Example B: P/D engines, session-aware
```bash
sgl-router --model-id qwen3 --chat-routing reorg \
--service-discovery --service-discovery-namespace inference \
--prefill-selector app=qwen3,role=prefill \
--decode-selector app=qwen3,role=decode \
--policy session_aware --session-id-header x-session-id \
--affinity-mode balanced --affinity-balanced-by running-requests \
--max-kv-usage 0.9
```
- **Engines:** Kubernetes discovery finds prefill and decode engines by label.
- **Policy:**
- Prefill engines use `session_aware`: requests with the same
`x-session-id` stick to the prefill engine they are bound to.
- Decode engines use `power_of_two`. To make decode session-aware too, use a
bucket file whose decode group sets `"policy": "session_aware"`.
- **Affinity:** in balanced mode the router samples an alternative engine, and
the session moves to it when the bound engine has more than 2x the
alternative's running requests **and** more than 4 more than it (the
defaults). The session also moves when the bound engine fails admission.
- **Admission:** prefill and decode engines are rejected at 90% KV usage.
### Example C: P/D buckets from a file
```bash
sgl-router --model-id qwen3 --chat-routing reorg \
--service-discovery --service-discovery-namespace inference \
--prefill-selector app=qwen3,role=prefill \
--decode-selector app=qwen3,role=decode \
--policy cache_aware \
--bucket-config buckets.json
```
`buckets.json` defines a short tier with a TTFT estimate and a long tier:
```json
{
"ttft_slo": "slo_first",
"buckets": [
{
"id": "short",
"max_input_tokens": 4096,
"max_context_tokens": 8192,
"ttft_ms": 300,
"prefill": {
"worker_services": ["inference/prefill-short"],
"admission": {"max_pending_prefill_tokens": 32768},
"affinity": {"mode": "balanced", "load_factor": 1.5}
},
"decode": {
"worker_services": ["inference/decode-short"],
"admission": {"max_kv_usage": 0.9}
}
},
{
"id": "long",
"rank": 1,
"max_context_tokens": 131072,
"prefill": {
"worker_services": ["inference/prefill-long"],
"policy": "power_of_two"
},
"decode": {"worker_services": ["inference/decode-long"]}
}
]
}
```
- **Engines:** each group takes the discovered engines of its role that belong
to the listed Kubernetes Services.
- **Buckets:**
- Requests that fit within 4k input and 8k context fit both buckets; `short`
is tried first because it is smaller. Longer requests go only to `long`.
- If `short` rejects a request, the router tries `long`.
- **Policy:**
- `short` prefill uses `cache_aware` from `--policy`.
- `long` prefill overrides it with `power_of_two`.
- Both decode groups use `power_of_two`.
- **Affinity:** `short` prefill uses balanced mode with a load factor of 1.5.
`power_of_two` groups take no `affinity` block; setting one is rejected at
startup.
- **Admission:**
- `short` prefill engines reject requests while 32k or more uncached tokens
are waiting.
- `short` decode engines reject requests at 90% KV usage.
- `long` engines set no limits.
-655
View File
@@ -1,655 +0,0 @@
# Engine selection: target design
This document describes the target architecture for engine selection. It defines
responsibilities and behavior; the interface and configuration examples are
sketches, not a specification of the current API or CLI. Implementation status is
listed at the end.
## Principles
1. **Filter buckets by token length, then order preferences.** `BucketResolver`
returns all compatible buckets, applying optional SLO preferences before
capacity/rank/ID ordering, without inspecting workers or policies.
2. **The bucket owns plain versus PD engine selection.** `Bucket::pick_engines`
calls its one plain group or both prefill and decode groups, returning a
complete selection. Both PD engines come from that same bucket.
3. **An engine group owns membership and its policy.** `EngineGroup::pick`
filters live workers by model, health, stage, and membership, invokes the
policy, and validates that its result belongs to the exact candidate set.
4. **A policy owns selection, fallback, admission, and its state dependencies.**
Construction injects the load, KV, or affinity handles it needs. Request-time
arguments contain facts and candidates. A policy cannot choose another bucket
or cross a PD role boundary.
5. **The handler owns bucket fallback and dispatch.** It calls `pick_engines`
on each bucket and dispatches only after a complete selection succeeds.
Missing candidates or admission rejection advance to the next bucket, where
all required engines are selected again. Invalid signals or policy results stop routing.
An engine is represented by `Worker`. Registry role labels remain authoritative
when filtering candidates. Groups reference worker IDs rather than owning live
workers. Group policy instances are reused across requests.
### Data ownership
| Type | Owns |
| --- | --- |
| `BucketResolver` | A model's bucket collection, length filtering, and ordering |
| `Bucket` | ID, length constraints, rank, groups, and complete plain/PD engine selection |
| `EngineGroup` | Engine membership, attached policy, and engine selection |
| `WorkerRegistry` | Live workers, model membership, health, and role |
| Request handler | Request preparation, ordered bucket attempts, HTTP errors, and dispatch |
Omitting both `worker_ids` and `worker_services` includes every healthy engine
serving the requested model and role. Explicit empty membership lists are rejected
by the config factory. `EngineGroup::new(policy)` creates a catch-all group.
`BucketGroups::Pd` requires both groups, making partial or mixed plain/PD bucket
configurations unrepresentable.
## 1. Code organization
```text
src/
buckets.rs Bucket resolution and engine-group selection
policies/
mod.rs Policy contract and construction
admission.rs Per-engine acceptance checks
cache_aware.rs Cache selection and local/remote adapter
session_aware.rs Session selection and fallback
sticky.rs Routing-key selection and fallback
least_load.rs Least-load selection
power_of_two.rs Pair sampling and stage-aware comparison
random.rs Random selection
round_robin.rs Rotation
state/
mod.rs Shared exports
kv_events/ Local index, subscriptions, hashing, wire format
load_monitor/ Engine reports and router-local request accounting
affinity_store.rs Assignments, expiry, and atomic updates
server/
app_context.rs Shared service lifecycle and policy wiring
routes/chat.rs Request preparation, PD coordination, dispatch
```
During migration, `buckets_reorg.rs` and `policies_reorg/` implement this design
beside the live `policies/` path. The layout above is the target after switchover.
Shared state already lives directly under `src/state/`.
Dependencies flow from engine groups to policies, and from policies and admission
to shared state. State does not depend on bucket ordering or concrete policy
strategies. Small shared helpers are sufficient; no generic score-composition,
tier executor, or separate selection framework is required.
## 2. Responsibilities and request flow
`BucketResolver::resolve(input_tokens, expected_peak_tokens, ttft_ms, tokens_per_second)` returns an ordered
list of compatible bucket references (possibly empty), or an invalid-signal error.
It does not receive a stage or a load view, resolve live engines, or invoke policies.
The handler iterates this list until a bucket supplies the complete engine selection.
`Bucket::pick_engines(workers, request)` accepts a `BucketRequest`
of prepared routing facts, invokes the required groups, and returns `BucketPick`
(one plain pick or a complete P/D pair). Failures retain their stage. This API
has no HTTP headers, `AppContext`, or forwarding dependency.
`EngineGroup::pick(workers, request)` resolves healthy workers for the request's
model and stage, intersects them with its membership, sorts them by stable ID,
and invokes its attached policy. It rejects foreign results, including a newly
allocated worker with the same ID as a candidate.
```text
chat_completions (reorg configured): prepare tokens and expected peak
|
v
BucketResolver::resolve: ordered length-compatible buckets
|
v
For each bucket: bucket.pick_engines(...)
|
+-- BucketGroups::Plain
| plain.pick() -> one plain engine
|
+-- BucketGroups::Pd
prefill.pick() -> P engine
decode.pick() -> D engine from the same bucket
|
+-- empty group / admission rejection -> try next bucket (repeat all picks)
+-- invalid signal / configuration / foreign pick -> return error
|
v
Complete selection -> forward_chat_request: acquire guards, attach PD bootstrap, forward response
```
The handler extracts token facts and header keys once into `BucketRequest`.
The bucket creates a stage-specific `PickRequest` for each group call, supplying
its own ID and the role associated with that group. Input length, expected peak,
token IDs, and session/routing keys pass through. Policies obtain observations
from their own shared-state handles; buckets and handlers do not provide load,
KV, or affinity services on each call. PD uses separate group policies, but never
independently resolves a decode bucket. Decode selection
failure discards that tentative prefill choice and advances to the next bucket
on missing candidates or admission rejection. No forwarding guards are acquired
and no prefill request is sent until both picks in one bucket succeed.
PD compatibility constraints beyond model and role remain follow-up work.
There is one endpoint: `POST /v1/chat/completions`. `AppContext::chat_routing`
chooses its implementation:
- `ChatRouting::Legacy` (default) uses the existing policies and bucket selector.
- `ChatRouting::Reorg(HashMap<ModelId, BucketResolver>)` uses the new bucket and
policy interfaces, with explicit model-specific resolvers.
`--chat-routing reorg` constructs default plain/PD buckets using the existing
`--policy` and tuning flags. Programmatic callers can still install explicit
resolvers. A missing model returns 404 without falling back to legacy routing.
The default serving path remains legacy.
Both implementations reuse request preparation (including sampling validation
and tokenization), forwarding, streaming, middleware, and the 32 MiB body limit.
The reorg implementation requests tokenization for length matching, retaining
the existing body-size estimate when tokenization is unavailable.
## 3. Bucket resolution
1. Validate that a known expected peak is at least the input length.
2. Keep buckets whose inclusive input-token range contains the input length.
3. Check the bucket context capacity against input plus requested output when
known, or against input length when the output budget is unknown.
4. Apply enabled SLO preferences to complete buckets. Count unmet preferences
equally, then sort by ascending input capacity (the lesser of the input upper
bound and context capacity), rank, and ID. Unbounded capacities sort last.
Return the entire ordered list, retaining nonpreferred buckets for fallback.
5. The handler calls each bucket's `pick_engines` until one supplies its complete
selection. A failed PD attempt never contributes an engine to a later pair.
The handler checks addition overflow when computing the expected peak.
An empty bucket list becomes a 400 `NoMatchingBucket` response. After exhausting
the list, accumulated admission rejection details produce a selection failure
(503); if there were no admission rejections, the last unavailable stage produces
a stage-specific 503. Invalid policy signals/configuration or out-of-candidate
picks stop the pass immediately with an internal error. Successful engine
selection ends the pass; forwarding errors do not restart bucket iteration.
Token ranges and rank belong to the bucket, not its engine groups. For a PD
bucket, both groups share this one request-length decision. Policy fallback on
a cache/affinity miss stays within that group's candidates. There is no second
pass with relaxed admission and no post-policy substitution.
### Optional SLO ordering
`Bucket` has optional `ttft_ms` and `tokens_per_second` estimates. The resolver
has independent `ttft_slo` and `tps_slo` preferences: `Disabled` (default),
`SloFirst` (matching first), and `BestEffort` (nonmatching first). The handler
parses `x-sgl-ttft-slo-ms` and `x-sgl-tps-slo` only when their preference is enabled;
invalid enabled headers return 400 before dispatch. Disabled headers are ignored.
SLO targets belong to bucket resolution, not the engine policy's `PickRequest`.
Absent targets are neutral. A bucket matches TTFT when its positive estimate is
at most the target, and throughput when its finite positive estimate is at least
the target. Missing or invalid estimates do not match a supplied target. Enabled
TTFT targets must be positive; throughput targets must be finite and positive.
Each unmet preference adds one ordering penalty. With both preferences set to
`SloFirst`, a bucket matching both comes before one matching either, followed by
buckets matching neither. Capacity/rank/ID breaks ties within these tiers. The
same logic applies to plain and PD buckets, and a PD bucket always supplies both
engines. TTFT and throughput preferences never independently resolve P/D groups.
Length constraints are applied first and admission rejection still advances to
the next complete bucket, including a bucket outside the preferred SLO tier.
Global session modes and sticky policies are follow-ups. Their
integration must preserve bucket-first selection and the same-bucket PD rule.
Cross-bucket affinity probing is not part of this interface. Session/routing
keys still pass through `PickRequest` for policies operating inside the selected
group; unsupported legacy modes need explicit migration decisions before the
standard serving path switches.
## 4. Policy and admission contracts
### Policy
All policies implement one asynchronous, object-safe `Policy::pick` interface.
A boxed future supports the remote prefix indexer; local policies can return an
immediately ready result.
```rust
pub trait Policy: Send + Sync + std::fmt::Debug {
fn pick<'a>(
&'a self,
engines: &'a [Arc<Worker>],
request: &'a PickRequest<'a>,
) -> BoxFuture<'a, Result<Pick, PickError>>;
}
```
Policies own their required state handles and read observations locally. Nested
fallback uses `pick_fallback(engines, request)`, which calls the fallback's `pick`;
the fallback reads its own state. There is no shared observation context or cache.
Buckets and HTTP handlers supply only candidates and request facts.
`Pick` identifies one engine and a selection reason for metrics and tracing.
`PickRequest` carries model, stage, selected bucket ID, input and optional
expected peak counts, optional token IDs, and session/routing keys. It contains
no HTTP body, bucket resolver, state handles, snapshots, or backend configuration.
| Outcome | Meaning |
| --- | --- |
| `Pick` | The chosen engine belongs to the supplied set and passed admission |
| `NoMatchingBucket` | No bucket supports the requested length |
| `NoCandidates` | No eligible member or policy selection miss |
| `NoAdmissibleEngine` | A policy exhausted its candidates; includes rejection reasons |
| `AdmissionRejected` | The chosen engine failed after-selection admission |
| `InvalidSignal` / `InvalidConfiguration` | Invalid policy input or configuration |
| `OutsideCandidates` | Policy returned an engine outside its exact candidate set |
### Admission
Admission evaluates acceptance. It does not rank engines, choose replacements,
change buckets, or mutate affinity.
An admission policy is a set of per-engine caps, `AdmissionLimits`. Each cap
is optional; an unset cap is not checked, and the default admits everything.
Each cap admits while the engine's load is below it. Usage caps are shares, in
(0, 1], of the engine's reported capacity, so one limit fits engines of any
size. The request itself is not counted: an engine just below a cap still takes
it, and the engine queues or retracts if it overflows. Admission observes load
without reserving it.
| Limit | Checks |
| --- | --- |
| `max_running_usage` | Running requests against reported max running requests |
| `max_kv_usage` | KV tokens against reported KV capacity |
| `max_waiting_requests` | Reported waiting requests |
| `max_pending_prefill_tokens` | Reported waiting uncached tokens |
| `max_inflight_requests` | Router-local in-flight requests |
```json
{"max_running_usage": 0.9, "max_kv_usage": 0.95, "max_inflight_requests": 64}
```
Unknown fields are rejected during deserialization.
The policy reads the selected engine's `EngineMetrics` from the load snapshot
it already captured for selection plus the live in-flight counter and calls
`EngineAdmission::check(engine, metrics)`, which returns `Allow`,
`Reject(limit name)`, or an error. Reported
metrics are `None` without a fresh, complete report, never zero, and such
limits fail open; the in-flight count is always known. Policies attach the
checker as `Arc<dyn EngineAdmission>` and decide where checking belongs in
their selection algorithm; there is no placement setting or filtering wrapper.
Power-of-two first selects an engine, then calls admission exactly once on that
engine. A rejection returns `AdmissionRejected` to the bucket loop; it does not
resample, choose the other sampled engine, or run a policy fallback. No candidates
returns `NoCandidates` without invoking admission. A single candidate is selected
directly; otherwise two distinct candidates are sampled uniformly, and the one
with lower stage pressure wins. A complete tie keeps the first sampled engine.
Power-of-two reuses the existing pure pressure-comparison functions. Plain and
prefill stages compare estimated prefill queue time when both reports provide it,
then waiting uncached tokens, waiting requests, and running requests. Decode
compares waiting requests, running requests, KV usage fraction, then used KV tokens.
Reported-pressure ties use router-local active requests. If either sampled engine
lacks a fresh, complete native report with valid capacity, both are compared by
router-local active requests instead. Basic reports from older publishers are
still passed to admission when fresh, but do not supply native pressure metrics.
Power-of-two retains the selected engine's load record from selection and
passes it to admission without another snapshot. A single candidate still has
its load read for admission, even though selection needs no comparison.
Neither the bucket nor HTTP handler supplies observations. Synchronous checks
do not fetch telemetry over the network themselves.
`AdmissionLimits::default()` is the default for new explicit policy attachments.
It leaves health, role, membership, and policy preferences in force. Migrated
configurations must retain their existing capacity and configured budget checks;
see compatibility below.
Affinity preferences never bypass attached hard admission.
Admission checks observe capacity; they do not reserve it. Concurrent requests
may pass against the same observation. Strict reservations would require a
separate mechanism.
## 5. Concrete policies
| Policy | Selection behavior |
| --- | --- |
| `RoundRobinPolicy` | Rotate over candidates using a cursor owned by this policy instance |
| `RandomPolicy` | Choose uniformly from candidates |
| `PowerOfTwoPolicy` | Sample two distinct candidates when possible and choose the lower-pressure engine using the stage's load comparison |
| `LeastLoadPolicy` (`load_based`) | Choose the least loaded engine; preserve tie-breaking, telemetry fallback, and recent-dispatch correction |
| `SessionAwarePolicy` | Prefer the session binding; use admitted power-of-two fallback and rebind |
| `StickyPolicy` | Reuse an admitted routing-key binding; use the configured fallback for new or missing keys |
| `CacheAwarePolicy` | Prefer an admitted prefix owner; use power-of-two fallback |
Session assignments are scoped by model, bucket ID, stage, and session key.
`SessionAwarePolicy::new(store, engine_load)` receives shared state; the caller
owns the store's idle timeout and eviction task. Missing or empty session keys
use power-of-two without creating assignments. A new or out-of-group binding
uses power-of-two with `AdmissionLimits::default()`, then the session policy
checks its selected engine before binding. A concurrent live assignment wins,
but is checked before returning it; rejection ends that attempt without
rewriting the binding or retrying another engine. Existing bindings follow the
shared affinity modes below. Session policies can be attached
independently to each role. Programmatic reorg callers configure
`model.affinity.session_id_header` for HTTP header extraction; this does not
enable legacy global modes or backup escape.
Session and sticky policies do not create assignments for missing keys. A
binding outside the candidates cannot win. A missing binding may invoke policy
fallback within the group. Rejected session affinity tries fallback without the
old engine; a successful admitted replacement rebinds the session. Failed fallback
preserves the old binding and advances to the next bucket.
Both affinity policies support `--affinity-mode prefer` (default) and `balanced`.
Prefer retains admissible affinity. Balanced compares it with an admitted
power-of-two alternative and switches only when the `--affinity-balanced-by` load
exceeds both the alternative times `--affinity-load-factor` (default 2) and the
alternative plus `--affinity-load-gap`. The load is `prefill-tokens` (default;
gap default 1024): native waiting uncached tokens plus the request's uncached
tokens on that engine, so the alternative also pays for the prefix the cache owner
holds. Session-aware has no prefix signal and counts the whole input on both
engines. `running-requests` (native or basic reports, gap default 4) ignores the
request. Both reports must be fresh; missing data
or ties preserve affinity. Bindings commit during selection after
admission, as with initial placement; dispatch failure does not roll them back.
Sticky fallback supports `round_robin`, `random`, `power_of_two`, and `load_based`,
with round-robin as the default. Nested fallbacks use
`AdmissionLimits::default()`; the owning policy explicitly checks the engine
returned by its fallback.
### Cache-aware behavior
The architecture preserves the cache algorithm within the supplied candidate
set. Its responsibilities are:
1. Look up prefix ownership through the local index or remote indexer.
2. Apply minimum matched-token and optional ratio thresholds.
3. Bound candidates using prefix/pressure ordering and the configured minimum,
ratio, and maximum worker counts.
4. Check admission and choose the best remaining prefix holder.
5. Apply the shared affinity mode. With no admitted prefix owner, fall back
within the group, excluding rejected engines.
Fallback samples two engines and checks the selected engine's admission; it
does not resample on rejection. Candidate limits apply only to prefix selection.
Memoize the prefix lookup once per request, including remote I/O. Each policy
restricts those matches to its own candidates. A memoized lookup does not imply
that another bucket or stage's cache selection and admission have already run.
Any optimization that skips those steps must establish that the previous result
applies to the current group.
## 6. Shared state and construction
Application wiring starts shared services once. Policy construction validates
configuration and passes the required handles to each policy and admission
implementation. For example, `PowerOfTwoPolicy::new(Arc<EngineReportedLoadTable>)`
retains the application's shared load table. KV-aware and affinity-aware policies
receive their corresponding shared handles when implemented. Policies with no
state dependency require none. Policy instances do not create duplicate
subscriptions, polling loops, indexes, or remote-client concurrency limits.
Requirements come from all configured policies, their admission checks, and
nested fallbacks. This includes tokenization, affinity-header extraction, load
observations, and dispatch timestamps. A role-group override must receive its
required settings even when the model's default uses a different policy.
### Load state
`state/load_monitor/` owns engine reports and existing router-local request
accounting. Power-of-two owns an `Arc<EngineReportedLoadTable>` and captures a snapshot
locally for each nonempty selection attempt. Its `pick` method selects the engine,
then passes that engine's borrowed load record directly to admission.
No snapshot or observation is added to `Pick`, `PickRequest`,
or the bucket interface, and no shared observation context is threaded through
policies or fallbacks.
The existing snapshot reader preserves rank aggregation, freshness, and capacity
fields. Missing, stale, or rank-incomplete reports yield `None`, not zero load.
A new pick reads current state. Admission reuses the selected observation even if
reports change after selection; it neither recaptures nor reserves capacity.
Fallback policies read their own state and do not share snapshots with callers.
Snapshot capture still scans the full table; an engine-scoped reader can be added
if profiling justifies it. Power-of-two ranks engines with the shared prefill/decode
comparisons in `state/load_monitor/engine_ranking.rs`, including router-local
fallback. Concrete load-aware admission remains
in #40271. Further shared load interpretation and correction for dispatches since
the report remain follow-ups; these must preserve source, freshness, and available
measurements without adding another
independent in-flight counter. Load and cache observations are not an atomic global
snapshot. Preserve request-guard cleanup.
### Cache state
`state/kv_events/` owns local subscriptions, event application, hashing, and the
radix-tree index. The tree stores prefix ownership and storage tiers, not KV
tensors. Eviction, invalidation, and worker removal update this shared state.
A small `CacheSource` adapter beside the cache policy normalizes local and remote
lookups into common prefix results. Preserve model/cache namespaces, required
rank information, and hash/block-size handling. Convert block counts to token
counts only with a valid conversion.
A confirmed miss and an unavailable backend both allow cache fallback, but
remain distinguishable in diagnostics. Invalid queries and configuration errors
remain explicit errors. Remote indexing and load-only deployments do not need
a duplicate local cache tree.
### Affinity state
`AffinityStore` owns scoped assignments, idle expiry, worker invalidation, and
atomic updates. Policies decide when to reuse or replace an assignment.
Concurrent first assignments must converge on an effective binding that is
still a candidate and passes admission. If a concurrent binding returns a
different engine, the policy must check that engine before returning it.
Reconcile conflicts with bounded retry.
Create or replace a binding only after admission succeeds. A binding records
preferred placement, not successful execution, so it may remain if later PD
selection or dispatch fails. It must not increment dispatch accounting.
## 7. Configuration and compatibility
Under `--chat-routing reorg`, `--bucket-config` reads a JSON file of complete
buckets. Each bucket sets `plain`, or both `prefill` and `decode`; each group may
set `worker_ids` or `worker_services`, plus `policy`, `admission` and `affinity`.
`affinity` overrides `mode`, `balanced_by`, `load_factor` and `load_gap` for a
session- or cache-aware group; unset fields take the CLI values.
```json
{
"ttft_slo": "slo_first",
"buckets": [
{
"id": "short-context",
"max_input_tokens": 4096,
"max_context_tokens": 8192,
"ttft_ms": 300,
"prefill": {"worker_services": ["inference/prefill-short"], "admission": {"max_pending_prefill_tokens": 32768}},
"decode": {"worker_services": ["inference/decode-short"], "admission": {"max_kv_usage": 0.9}}
},
{
"id": "long-context",
"rank": 1,
"max_context_tokens": 131072,
"prefill": {"worker_services": ["inference/prefill-long"]},
"decode": {"worker_services": ["inference/decode-long"]}
}
]
}
```
A request with 4k input tokens and a 16k expected peak cannot fit the short
bucket's context capacity. It selects the long bucket and both of its P/D groups.
With a known peak of 8k or less, the same input selects both groups of the short bucket.
Membership has two mutually exclusive forms. `worker_ids` pins exact discovery
identities: URLs for static discovery, `namespace/pod-UID` for Kubernetes pods.
`worker_services` lists Kubernetes `namespace/service` names and matches any
listed Service through its EndpointSlices' `kubernetes.io/service-name` label.
Service membership survives pod replacement and scale-up; worker IDs still
change per pod incarnation so breaker, load and affinity state cannot carry over.
Only Services included by the discovery watch are visible. Multiple slices and
Services selecting one pod contribute the union of its memberships; removing a
slice removes only memberships no longer advertised by another slice. Membership
updates preserve the live worker and its in-flight state. Readiness and routing
use the same group filter, including model, health and role constraints.
Omitted group fields mean every engine of the group's role and `--policy`
(power-of-two on decode). Admission limits a group leaves unset take
`--max-in-flight` / `--max-kv-usage`, field by field. JSON `null` also inherits;
there is no per-group disable value. To limit only some groups, leave the CLI
default unset and configure those groups explicitly. A group policy is
`power_of_two` or the `--policy` kind, whose affinity, cache and tokenizer
settings are the ones resolved. Startup rejects empty or duplicate IDs, a bucket
that is not exactly plain or P/D, inverted token ranges, a minimum input above
the context capacity, empty membership lists, blank worker IDs, malformed
Service names, both membership fields together, non-positive capacity or SLO
estimates, usages outside (0, 1], zero count limits, and a policy a stage cannot
serve. Without the flag, reorg builds one plain and one P/D bucket
over all engines. The worker registry still rejects mixed plain and PD engines
within one model.
Legacy `--bucket-config` files describe role-specific buckets and do not
translate: P/D associations must be written explicitly, never inferred from
rank or names.
Retained settings keep their meanings, defaults, units, and validation unless a
change is listed below. Policy-specific tuning applies to role groups using that
policy. Reject unsupported settings and incompatible combinations at startup;
do not accept and ignore them.
### Retained behavior
- Keep `load_based` as the CLI name for `LeastLoadPolicy`.
- Preserve explicit engine membership and context constraints. Bucket-level
ranges and ordering replace independent per-stage selection.
SLO preferences order whole buckets; deployment configuration remains explicit
until the production configuration factory and serving switchover are ready.
- Preserve cache-provider selection, endpoint validation, query timeout and
concurrency limits, and unavailable-backend fallback.
- Preserve the 1,024-token default minimum cache hit, optional ratio gate,
and candidate bounds. Shared affinity modes replace cache diversion knobs.
- Preserve session and sticky headers, idle timeouts, eviction cadence, and the
four sticky fallback choices. Global modes need a bucket-first migration design.
- Map `--filter overloaded` and `--max-in-flight` to `max_inflight_requests`;
the existing router-local counter remains the source. `--max-kv-usage` sets
`max_kv_usage` for groups that leave it unset.
- Preserve configured capacity, pending-prefill, and in-flight checks, including
their missing-report behavior. Power-of-two applies admission to its selected
engine; other policies explicitly place checks in their selection logic.
Other paths use `AdmissionLimits::default()` unless a check is configured.
Never silently discard a configured budget.
Listener and shutdown configuration, discovery, worker health and circuit
breakers, tokenizer loading, request timeouts, sampling overrides, and logging
remain outside the policy redesign. Preserve their existing behavior and
validation, including dispatch-time breaker probes and request cancellation.
### Deliberate changes
| Behavior | Target |
| --- | --- |
| Power-of-two admission | Check only the chosen engine; rejection advances to the next bucket |
| Round-robin cursor | One cursor per role-group policy instance |
| Capacity exhaustion | Try the next compatible bucket; return accumulated rejection details if all fail |
| Primary/backup proposals and post-policy substitution | Removed; each policy returns one engine |
| Session affinity | Reuse or replace bindings through shared affinity modes |
| Omitted `--affinity-mode` | Defaults to `prefer` for both affinity policies |
| Pressure-guard tuning | Replaced by `--affinity-load-factor` and `--affinity-load-gap` |
| Policy attachment | Explicit role-group policy overrides the applicable model/stage default |
Reject these dropped options explicitly:
- `--policy fused_score` and `--policy score_policy`, including `--fuse` terms
and weights.
- `--decode-policy legacy_host_affinity`.
- `--stable-pair`.
- `--affinity-mode soft` / `strict`; use `prefer` / `balanced`.
- Legacy pressure guards, cache switch margins, queue limits and saturation floors.
- `--filter prefix_cache` and `--prefix-cache-min-share`. The removed prefix-share
filter is not equivalent to the cache-aware minimum-hit gate.
## 8. Dispatch and failure handling
A successful pick means admission passed against the observed state. Health and
capacity can change before dispatch. The handler owns network operations, retry
rules, and accounting for the engines actually dispatched to.
Selection fallback advances through the ordered compatible buckets before any
network dispatch. Every PD attempt picks both engines from that bucket. With
`--retry-max-attempts` above 1, a dispatch that fails before any response
reaches the client reruns selection from the first bucket with the failed
engine excluded (`BucketRequest::excluded`); it never replaces a pick whose
response the client has started to receive.
For PD, acquire and release accounting for the actual stages and clean up
partial setup on failure. Policy selection does not own the PD request lifetime.
Selection metrics must not imply that dispatch or execution succeeded.
## Implementation status
This PR adds the side-by-side interfaces in `src/buckets_reorg.rs` and
`src/policies_reorg/`, plus a configurable bucket-first implementation behind
`chat_completions`. The live `src/policies/` path remains the default.
Implemented here:
- `BucketResolver::resolve` returns all length-compatible buckets in optional
SLO-preference tiers, with capacity/rank/ID order within each tier.
- `Bucket::pick_engines` owns plain/PD orchestration and stage-specific policy
requests; `BucketRequest` carries prepared facts and `BucketPick` retains picks.
- `Bucket` owns input limits, context capacity, rank, and plain-or-PD groups.
- `EngineGroup::pick` owns live candidate filtering, policy invocation, and
exact candidate validation, without cross-bucket fallback.
- `Policy::pick`, within-group fallback interface, per-engine `EngineAdmission::check`,
and `AdmissionLimits` with running and KV usage shares plus waiting,
pending-prefill and in-flight counts. Power-of-two samples two distinct engines, compares stage pressure,
and checks its selected engine with no replacement on rejection.
- Policy-owned load dependency and local observations. Power-of-two passes the
selected engine's load record directly to admission, without another snapshot.
`PickRequest`, `Pick`, and bucket APIs carry no load observations.
- The reorg chat implementation iterates resolved buckets, calls `pick_engines`,
advances on empty candidates/admission rejection, and
dispatches only after one complete selection. Exhaustion retains admission reasons.
- `AppContext::chat_routing` configures legacy versus reorg routing on the same
endpoint and carries the reorg model-resolver map.
- Optional bucket TTFT/throughput estimates and preferences, with enabled-header
parsing and whole-bucket fallback in the shared chat route.
- `CacheAwarePolicy` reads local radix-tree or remote indexer prefixes, intersects
exact worker URLs with the current group, applies hit thresholds and candidate
bounds, and applies the shared affinity mode.
- `PrefixMemo` shares lookup results (including misses and unavailable backends)
across bucket attempts for one prepared request. Entries are keyed by the shared
`Arc<CacheSource>` so different index namespaces remain independent. Each pick
reruns its own candidate filtering and admission after obtaining a fresh snapshot.
- Cache selection checks bounded candidates explicitly; rejected owners are
excluded from fallback. The cache policy checks its fallback winner's admission.
Cache policies require plain/prefill groups.
- `SessionAwarePolicy` reuses admitted model/bucket/role-scoped bindings from a
shared `AffinityStore`, falling back to power-of-two for new or keyless sessions.
Assignments follow admission; concurrent binding winners are rechecked.
Rejected affinity invokes fallback; failed fallback preserves the binding.
The caller owns expiry and sweeper lifecycle. A binding may remain after a
later PD group fails, because it records placement rather than dispatch.
- `--bucket-config` builds explicit plain and P/D buckets with per-group
membership, policy and admission; omitting it builds the default buckets.
Follow-up work includes remaining selection policies.
Not yet implemented in the reorg path:
- Other concrete selection policies.
- Global session modes and sticky routing-key affinity.
- Power-of-k cache-miss fallback configuration and cache decision metrics.
- Shared load interpretation, dispatch correction, and policy-specific
dispatch-timestamp requirements.
- PD compatibility filtering and legacy-route switchover.
The preceding policy sections describe target behavior for those follow-ups;
they do not claim those capabilities are present in this PR.
File diff suppressed because it is too large Load Diff
+640
View File
@@ -0,0 +1,640 @@
# sgl-router reference
Detailed behavior of the features summarized in [README.md](README.md). Routing
design, policies and bucket configuration are in [ARCHITECTURE.md](ARCHITECTURE.md).
- [Discovery and KV signals](#discovery-and-kv-signals)
- [Bucket file membership](#bucket-file-membership)
- [Routing and reliability](#routing-and-reliability)
- [Chat rendering](#chat-rendering)
- [Native `/generate`](#native-generate)
- [OpenAI over /generate](#openai-over-generate)
- [Embeddings](#embeddings), [Classify](#classify), [Completions](#completions), [Rerank](#rerank)
- [Anthropic Messages](#anthropic-messages)
- Per-model rendering: [DeepSeek V4 and V4.1](#deepseek-v4-and-v41), [GLM-5.3](#glm-53), [Qwen3.8](#qwen38), [Kimi-K3](#kimi-k3)
- [HTTP/2](#http2)
- [Native gRPC](#native-grpc)
- [Upgrading from `cache_aware_zmq`](#upgrading-from-cache_aware_zmq)
## Discovery and KV signals
### PD version groups
To run two engine versions side by side in PD mode (e.g. during a rollout),
pass `--pd-version-group-label <key>`. A prefill worker is then paired only with
decode workers that have the same value for that EndpointSlice label (inherited
from the Service, so use one Service per role and version), so KV never crosses
versions. Unlabeled workers form their own group.
```bash
sgl-router --model-id qwen3 --service-discovery \
--prefill-selector app=engines-qwen3,role=prefill \
--decode-selector app=engines-qwen3,role=decode \
--pd-version-group-label sglang.ai/version-group
```
### External KV indexer
```bash
sgl-router \
--model-id qwen3 \
--tokenizer-path /models/qwen3/tokenizer.json \
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
--policy cache_aware \
--cache-prefix-provider indexer \
--kv-indexer-endpoint http://10.0.0.10:50051 \
--kv-indexer-query-timeout-ms 100 \
--kv-indexer-query-max-inflight 32
```
The Indexer replaces the Router-local radix tree as the native Cache-Aware
signal. Query timeouts and local concurrency are bounded by the two Indexer
options, which default to 100 ms and 32 respectively.
### LoRA cache identity
LoRA routing requires workers that publish `BlockStored.lora_name`. Requests
select the adapter through `model: "base:adapter"` on OpenAI routes, or
`lora_path` on native routes. Salted generation requests also carry `cache_salt`.
Worker block IDs are opaque and may differ across adapter loads or workers.
The router derives adapter-aware lookup keys from event tokens and keeps a
per-worker, per-DP-rank mapping for parent links and removals. Multiple block
IDs can own the same lookup key; removing one leaves the other owners and
storage tiers intact. The remote indexer bridge performs the same translation.
Base-model events retain their existing published-key path.
New replicas request `?include_lora=true` snapshots containing LoRA mappings
(version 2). Legacy consumers receive a base-only version-1 view, so mixed
snapshots do not prevent base-cache bootstrap during rolling upgrades. Base-only
snapshots retain version 1 for every consumer. LoRA stores buffered during bootstrap retain their
tokens; the original base-model buffering and replay behavior is unchanged.
Reloading different adapter weights under the same public name can still
credit stale engine blocks until their eviction. Names identify adapters;
versioned names are needed to distinguish different weights.
### KV-event gap replay
ZMQ drops KV events at the publisher's high-water mark, and a lost removal
leaves the router crediting a worker with blocks it no longer holds. Give each
engine a replay socket and the router re-fetches the batches a sequence gap
skipped:
```bash
python -m sglang.launch_server ... \
--kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558"}'
```
Rank r publishes on 5557 + r and replays on 5558 + r, so with `dp_size > 1`
place the replay base at least dp_size ports past the PUB base. The engine keeps
the last `buffer_steps` batches (default 10000); older gaps stay unrepaired.
`sgl_router_kv_event_replays_total{outcome}` counts the attempts.
### Pending prefixes
Until the engine's KV events arrive, requests that share a cold prefix see no
cached owner and spread across workers, each prefilling the same prefix. This
is common with parallel sampling, RL rollouts and agent fan-out.
`--cache-pending-prefix-ttl-ms` credits a worker with a prompt's prefix for that
long after routing it there, so the burst stays together. It is off by default
and needs `--policy cache_aware` with the Router-local radix tree.
`sgl_router_cache_pending_prefix_hits_total` counts lookups where a pending
prefix matched deeper than any confirmed one; the policy's candidate and
admission filters still decide whether that worker is picked.
### Peer bootstrap (Kubernetes)
A replica that starts mid-fleet subscribes to each worker's KV topic
mid-stream, so everything already resident in the engines' caches is invisible
to it and it routes cache-blind until traffic re-stores those blocks —
degrading the engines' locality for the whole fleet, once per replica on every
rolling update. With a peer selector set, a booting replica instead pulls a
tree snapshot from a warm sibling over `/internal/kv_snapshot` and splices it
under its own live delta stream, and `/readyz` stays 503 until that bootstrap
settles. Off unless `--kv-peer-selector` is set; requires `--policy
cache_aware` with the Router-local radix tree (not an external Indexer) and
`--service-discovery`, which the CLI enforces.
```bash
sgl-router \
--model-id qwen3 --policy cache_aware \
--service-discovery --service-discovery-namespace prod \
--selector app=engines-qwen3 \
--kv-peer-selector kubernetes.io/service-name=sgl-router
```
- `--kv-peer-selector` is matched against **EndpointSlice labels** — the
router Service's labels plus `kubernetes.io/service-name`, never the pods'
labels, so `kubernetes.io/service-name=<router-service>` is the unambiguous
choice. A pod-template label matches nothing and every replica boots cold.
- `--kv-bootstrap-timeout-ms` (default 600000) bounds the whole bootstrap;
readiness waits on it, so startup/readiness probes must tolerate a replica
staying unready this long.
- `--kv-bootstrap-fetch-timeout-cap-ms` (default 300000) caps one snapshot
fetch within that budget.
- `--kv-bootstrap-seed-required` (off by default) keeps holding `/readyz` when
siblings were present but their tree could not be pulled, bounded at
max(3× the bootstrap timeout, 60s) — it delays a failed seed's replica (and
with it a rolling update) rather than shipping it cache-blind; it cannot
stall a rollout indefinitely.
The deployment needs two things beyond the flags: the router's ServiceAccount
must hold `get`/`list`/`watch` on `endpointslices` in the watched namespace
(the worker-discovery Role already grants this), and the pod spec should wire
`POD_NAME`, `POD_NAMESPACE` and `POD_IP` from the downward API so a replica
can exclude itself from its own peer list. A rolling update should surge
(`maxUnavailable: 0`) so new replicas always have warm siblings to copy from.
`tests/e2e/k8s_integration/manifests/kv-bootstrap.yaml` is a worked example of
all of it, and the `sgl_router_kv_bootstrap_*` series in
[monitoring/README.md](monitoring/README.md) make the whole path observable.
## Bucket file membership
`worker_services` matches Kubernetes Services by `namespace/name`, using the
`kubernetes.io/service-name` label on watched EndpointSlices. Replacement pods
and new replicas join the same group automatically. The router's discovery
selectors must include those EndpointSlices; this field does not expand the watch.
A worker selected by several Services belongs to each of them.
For static URL discovery, use `"worker_ids": ["http://worker:30000"]`: each ID
is the configured worker URL. Kubernetes worker IDs are `namespace/pod-UID`
(and change when a pod is replaced), so use `worker_services` for durable pools.
Set only one membership field, or omit both for every engine of the group's role.
Empty membership lists and blank worker IDs are rejected at startup.
Admission fields left unset or set to `null` inherit CLI defaults. To apply a
limit only to selected groups, omit that CLI default and set it on those groups.
## Routing and reliability
### Optional tokenizer for load-only routing
`--no-tokenizer` skips tokenizer loading for load-only policies such as
`power_of_two` and `session_aware`, on either routing path. Workers tokenize the
original messages, and `/v1/tokenize` and `/v1/detokenize` are unavailable.
Cache-aware routing, prefix-cache terms or filters, and buckets with token-length
or context limits require a tokenizer. Reorg buckets that only select membership
and load-based policies can use `--no-tokenizer`.
### DP-rank routing
An engine launched with `--dp-size` or `--attn-dp-size` runs several DP ranks,
each with its own KV cache, behind one endpoint. With `--dp-aware`, the router
also picks the rank inside the selected worker. It sends that rank as
`X-Data-Parallel-Rank`, which the engine honors, and overwrites any value the
client sent. The router picks the first of these that applies:
1. A hash of the sticky routing key or session id, so a conversation keeps
its rank and router replicas agree.
2. The rank with the deepest cached prefix in the local KV tree.
3. The rank with the fewest requests this router has in flight on it, or with
`--dp-rank-policy round_robin`, the worker's ranks in turn.
In PD mode, decode is ranked by step 3 only. The decode request names the
prefill rank in `disagg_prefill_dp_rank`, and the bootstrap room satisfies
`room % prefill_dp_size == prefill_rank`.
### Retries
`--retry-max-attempts N` (default 1, which disables retries; 3 is typical) lets
a request that fails before any response reaches the client be sent again to a
worker it has not tried yet. A failure is a transport error, an open circuit
breaker, a 5xx, a 429, or a timeout: `--request-timeout-secs` bounds a whole
non-streaming response, and a streaming one until its headers arrive. SGLang's
chat endpoint sends those headers with the first token, so for streaming chat
the same flag bounds time to first token, queueing included; raise it for
long-context or deeply queued fleets. Once a streaming response's 2xx status
has been sent, it is never retried, and neither is any other 2xx or 4xx. In PD mode the failed
side is excluded, or both sides when the caller set `rid`, which an engine still
running it would refuse; a new pair gets a new bootstrap room. Retries share the
request's `--stale-request-timeout-secs` deadline, and none starts after it. When
every eligible worker has failed, the client gets the last failure.
`sgl_router_retries_total` counts the retried attempts.
Each retry first waits out a jittered exponential backoff: a random delay in
`[d/2, d]`, where `d` starts at `--retry-initial-backoff-ms` (default 50),
doubles per retry, and is capped at `--retry-max-backoff-ms` (default 2000).
Set the initial backoff to 0 to retry immediately. A backoff never waits past
the request's stale deadline.
### Engines with `--api-key`
The router reads each worker's `/server_info` and `/model_info` to learn its
model, KV-event publisher, HTTP/2 support and DP size. An engine launched with
`--api-key` rejects those requests without the key, so pass the same key as
`--worker-api-key`. Chat requests and `/flush_cache` forward the caller's
`Authorization` header instead, so callers still need the engine key.
### Fleet-wide sampling contract
`--override-sampling-params` fixes the sampling configuration for every client
of this router, independently of what the engine's own defaults happen to be
(on native [`/generate`](#native-generate), in each prompt's `sampling_params`):
```bash
sgl-router \
--model-id qwen3 \
--tokenizer-path /models/qwen3/tokenizer.json \
--worker-urls http://10.0.0.1:30000 \
--override-sampling-params '{"temperature": 1, "top_p": 0.95, "n": 1}' \
--sampling-param-conflict reject
```
It takes one JSON object keyed by the request-body field names
(`temperature`, `top_p`, `top_k`, `min_p`, `repetition_penalty`,
`frequency_penalty`, `presence_penalty`, `n`). A configured value is injected
whenever the request omits that field, so the engine's defaults cannot drift
from what the operator declared. `temperature`, `top_p`, `top_k`, `min_p` and
`repetition_penalty` are the five the engine resolves from the model's own
`generation_config`, which is what makes them drift when a deployed image
changes; the rest have fixed API defaults.
An explicit `null` counts as omitting the field, not as a client-supplied
value: the OpenAI API types these parameters as nullable with a documented
default, so `null` asks for the default — and on a governed fleet the
configured value is what the default is.
For a request that does send a value, `--sampling-param-conflict` decides:
`reject` (the default) 400s a differing value before admission, while `allow`
forwards the client's value untouched — the router never silently rewrites what
a client sent. A `reject` response carries
`x-router-error-code: sampling_contract_violation` and is counted in
`sgl_router_sampling_contract_rejections_total{param}`, so a rollout's blast
radius is visible per parameter rather than folded into `bad_request`.
`reject` also 400s a value it cannot read as a number, on a governed parameter
only. The engine coerces more than JSON numbers — a bool and a numeric string
both become numbers — by rules that are undocumented and need not match across
a fleet, so a value the router cannot read is one it cannot prove conforms, and
waving it through would make the pin bypassable. The common coercions are
matched exactly (`false` is 0, `"0.5"` and `"1_0"` are 0.5 and 10), so this
refuses only genuine garbage. `allow` is unaffected: it promises nothing, so
such a value keeps flowing to the engine, which owns the request schema.
A value may also be an inclusive band, `{"min": LO, "max": HI}`, for a
parameter that stays tunable inside a range. A band names no value to inject,
so it constrains only the requests that name the parameter; one that omits it
gets the model's own `generation_config` default, which the router cannot see.
Because a band can only ever reject, combining one with `allow` is a startup
error.
Values are range-checked at startup, so a misconfiguration fails the launch
instead of 400ing every request at the engine. Repeating a key in the flag is
also a startup error, rather than silently enforcing whichever copy came last.
| parameter | accepted | notes |
| --- | --- | --- |
| `temperature` | `[0, 2]` | |
| `top_p` | `(0, 1]` | |
| `top_k` | `>= 1`, or exactly `-1` | `-1` is the engine's "whole vocabulary" spelling and its default. Being non-contiguous it cannot bound a band. Note `top_k: 1` is greedy decoding, not "disabled". |
| `min_p` | `[0, 1]` | not an OpenAI parameter; the engine's domain |
| `repetition_penalty` | `(0, 2]` | not an OpenAI parameter; the engine's domain |
| `frequency_penalty` | `[-2, 2]` | |
| `presence_penalty` | `[-2, 2]` | |
| `n` | `[1, 128]` | |
The OpenAI domains are deliberately narrower than what the engine itself
accepts (`SamplingParams.verify` would take `temperature: 5`): these values are
injected into request bodies, and a fleet contract outside the range every
OpenAI client library validates against is far more likely a typo than an
intent.
Cost: a request that named every configured field forwards its original bytes
untouched. One that omits a field has the scalars spliced directly into the
body bytes — no parse, no re-serialize — which on a 16 MiB body is ~0.24 ms
against ~5.7 ms for a `serde_json` round-trip. Only `input_ids` and PD
bootstrap injection still parse and re-serialize, because those may have to
overwrite a key the client sent.
### Relationship to the engine's own `--preferred-sampling-params`
The engine has an inject-when-absent flag of its own,
`--preferred-sampling-params`, merged in
`python/sglang/srt/managers/tokenizer_manager.py` as
`{**preferred, **obj.sampling_params}`. On `/v1/chat/completions` it is
currently a no-op: `ChatCompletionRequest.to_sampling_params`
(`python/sglang/srt/entrypoints/openai/protocol.py`) resolves every sampling
key eagerly through `generation_config` and then its own defaults, so the
right-hand side of that merge is always fully populated and always wins.
(`/v1/responses`, in the same file, already omits `None` entries for exactly
this reason.) If that is fixed engine-side, `--preferred-sampling-params`
covers the inject-when-absent half for a single engine.
What stays the router's to own either way is the enforcement half: the
`reject` immutability contract with its 400 before admission — costing no
queue slot and no engine round-trip — the `sampling_contract_violation` code
and per-parameter counter, bands, and one contract applied at a shared ingress
across engines whose own flags the router operator may not control.
## Chat rendering
The router renders chat requests with dynamo-render (`dynamo-renderer`): the model's
HF Jinja template from `tokenizer_config.json` or a sibling
`chat_template.jinja`, or dynamo-render's built-in DeepSeek encoder (V4 family, V3.2)
for template-less models. Cache-aware routing hashes the rendered tokens so its
prefix queries match the blocks the engine caches. Models the engine encodes in
code but dynamo-render cannot tokenize here (Inkling) route via raw prompt
text, as does any model whose template fails to load or render.
Some chats additionally forward the rendered tokens to the engine as
`input_ids`, retaining the original messages, so the engine skips
re-tokenizing. How many depends on the model's renderer. DeepSeek-V4's, V4.1's
and Kimi-K3's native encoders and GLM-5.3's and Qwen3.8's templates are
fixture-verified against SGLang's request normalization, so they forward every
chat except multimodal ones and those with caller-provided `input_ids`. Renderers
without that verification (HF Jinja templates)
forward only plain text chat requests (string `content`, no tools, no template
kwargs or reasoning controls or historical `reasoning_content`, no assistant
continuation, no consecutive users or non-leading system turns) and warn
`UNVERIFIED` at startup; every other request shape is rendered for routing
only.
Use matching model files on the router and workers, and set
the same `--default-chat-template-kwargs`, `SGLANG_DEFAULT_THINKING`,
`SGLANG_DSV4_REASONING_EFFORT`, and `SGLANG_DSV41_REASONING_EFFORT` on both.
The router reads these render defaults from its own configuration and environment;
it does not discover the workers' settings. Point `--tokenizer-path` at the workers'
model snapshot so the V4 effort profile is read from the same
`encoding/encoding_dsv4.py`.
Set `--disable-input-ids-forwarding` for this router's model when worker-side
rendering has not been verified to match. This disables router-generated IDs
for every routing policy; cache-aware routing still renders and tokenizes
locally, and the original messages reach the workers for engine processing.
Caller-supplied `input_ids` remain caller-owned and pass through unchanged.
Forwarding logs its assumptions at startup. In particular, disable it when the
router's render defaults differ from the workers', for worker template overrides
not reflected in the router's model files, for worker parser overrides such as
`--tool-call-parser deepseekv32` that select a native encoder over a shipped
template, or conversation templates with stop strings (the engine's `input_ids`
path skips those template stops). Disabling forwarding preserves engine behavior
but does not establish parity for local routing hashes.
Also set `--disable-input-ids-forwarding` for array-only templates: Dynamo may wrap
string content into arrays differently from the worker. The pinned Dynamo renderer does not expose
its conversion flag, so the router cannot automatically block these templates.
Detailed content-format parity coverage follows in #39133.
The Dynamo crates are pinned exactly and `Cargo.lock` is committed; CI builds
with `--locked`, so rendered bytes cannot change without a reviewed diff.
Router tokenization sits on the TTFT path for every chat it renders. Two opt-in
flags make it cheaper:
- `--tokenizer-backend fast` encodes with fastokens (decoding stays on HF). It
needs a `tokenizer.json` and falls back to `hf` when fastokens cannot load it.
- `--tokenizer-l1-cache-mb N` caches prefix tokenizations at special-token
boundaries, so a multi-turn chat encodes only the turns added since the
previous request. Boundaries are unconditional, non-normalized, non-stripping
special tokens with no overlapping added-token spellings. Unsafe candidates
are excluded; if none remain, encoding proceeds without the cache.
On a ~69K-token DeepSeek-V4 chat, `hf` encodes in ~40 ms, `fast` in ~4 ms, and
a new turn on a cached history in ~0.2 ms. The DeepSeek fixtures check every
case under `hf`, `fast`, and `fast` with L1. Startup logs report the resolved
backend and cache state. `/metrics` exposes only
`sgl_router_tokenizer_l1_tokens_total{source="cached"|"encoded"}` to measure
how much tokenization work the cache reuses.
## Native `/generate`
`/generate` (`POST` or `PUT`) has the engine's interface: the same
`GenerateReqInput` body and the same response, buffered or SSE. The body names no
model, so requests go to the one this router serves. Worker selection
(`--chat-routing`, `--policy`), PD dispatch, and abort-on-disconnect are shared
with chat completions.
With a tokenizer loaded, the router tokenizes `text` (a string or a list) with
the special tokens SGLang adds (BOS per `add_bos_token` for Llama-, Gemma- and
Cohere-class tokenizers, otherwise the `tokenizer.json` post-processor's), and
forwards it as `input_ids`. The engine skips tokenizing and routing sees its exact
tokens. Multimodal requests keep `text`, since the engine expands placeholders
from it, and `--disable-input-ids-forwarding` keeps it for every request. So does
a model whose `tokenizer.json` normalizer transformers replaces on load (legacy
SentencePiece Llama files, bge-m3), or whose `tokenizer_config.json` the router
cannot read, as the router cannot reproduce its tokens. A batch goes to one
worker: load counts every prompt and each of its `n` samples, while bucket and
context limits bound the longest prompt plus its own `max_new_tokens`.
Otherwise the body passes through, plus PD bootstrap fields, a minted `rid` for
a single prompt that has none, and `--override-sampling-params` defaults. Under
`--dp-aware` each worker's body also carries the chosen `routed_dp_rank`; a PD
batch or `n > 1` request leaves the prefill rank to the engine, which gives item
`i` the bootstrap room `room + i`.
## OpenAI over /generate
`/v1/completions` and `/v1/chat/completions` behave as the engine's own routes,
but the router runs SGLang's OpenAI layer itself (`sglang-processor`'s `openai` module): it lowers
the request to the `GenerateReqInput` SGLang would build, sends it to
`/generate` like a native request, and builds the OpenAI response, buffered or
SSE, from the engine's output. The engine's `/server_info` supplies the server
args that layer reads (`--enable-cache-report`,
`--stream-response-default-include-usage`, `--incremental-streaming-output`, the
custom-labels header). Requests it cannot reproduce exactly go to the engine's
own route as sent: workers that disagree on those args or never
reported them, `--completion-template`, `return_hidden_states` and other
`sglext` outputs, `echo` or `logprobs` without a router tokenizer, and bodies
SGLang would reject. Chat also needs a renderer verified against SGLang (the
`AllText` scope: DeepSeek-V4, V4.1, GLM-5.3, Kimi-K3 and Qwen3.8 today), router `--default-chat-template-kwargs`
equal to the workers', a reasoning parser the processor reproduces
(`deepseek-v4`, `deepseek-v41`, `glm45`, `kimi_k3`, `qwen3`) or none, and no media yet. Tools are served with `tool_choice`
`auto` or `none`, non-strict, with standard JSON-schema types, and a tool
parser the processor reproduces (`deepseekv4`, `deepseekv41`, `kimi_k3`, `qwen3_coder`); `required`, named and strict
tools take a constraint from the engine's xgrammar, so they go to the engine
(`kimi_k3`'s `auto` tag is reproduced).
With `--tool-call-parser glm47`, a chat with tools also goes to the engine: its
grammar needs `ebnf_full_assistant`, which `/generate` does not accept.
The router reads the model's `config.json` and `generation_config.json` beside
`--tokenizer-path` (or from the HF repo), as the engine reads them, so a local
path should point into the model's directory. Engine env vars that change the
layer (`SGLANG_TOOL_STRICT_LEVEL`, `SGLANG_FORWARD_UNKNOWN_TOOLS`) are taken as
unset. `sgl_router_openai_route_total` counts each outcome.
## Embeddings
`/v1/embeddings` has the engine's interface: the same OpenAI `EmbeddingRequest`
body and response. As for chat completions, `model` must name the served model.
A PD fleet answers 400, since prefill and decode engines serve no embeddings.
Text `input` (a string or a list) is tokenized and forwarded as token IDs, as for
[`/generate`](#native-generate), plus the EOS SGLang appends for EmbeddingGemma.
Blank prompts and multimodal items stay as sent, for the engine to reject or
render. A list is a batch for one worker, routed on load. A single prompt with no
`rid` gets a minted one. Under `--dp-aware` the engine picks the rank, since its
embeddings endpoint reads none.
## Classify
`/v1/classify` has the engine's interface: the same `ClassifyRequest` body and
response. It is served as embeddings are, except that a batch of text stays text,
since the engine takes token IDs for only one prompt.
## Completions
`/v1/completions` has the engine's interface and is prepared like chat (sampling
contract, PD bootstrap, DP rank), with the prompt forwarded as sent: tokens the
router computes for routing do not replace it.
## Rerank
`/v1/rerank` has the engine's interface: the same `V1RerankReqInput` body and
response, sent as `POST` or `PUT` to the model this router serves. The body is
forwarded as sent, since the engine renders and tokenizes each query-document
pair. Routing is on load, with each pair's size estimated from its text. As for
embeddings, a PD fleet answers 400 and `--dp-aware` pins no rank.
## Anthropic Messages
`/v1/messages` converts each request to a chat completion at ingress, so it is
routed exactly like `/v1/chat/completions`, and converts the reply (or stream)
back to the Messages format. Consecutive turns of the same role are merged, as
the Messages API does, and a final assistant turn is sent as a prefill
(`continue_final_message`). Besides the Messages fields, only
`chat_template_kwargs` and `continue_final_message` reach the chat request.
Every error uses the Anthropic error envelope; 5xx messages are replaced with a
generic one, and a failed stream still closes its open block and ends with
`message_stop`.
`/v1/messages/count_tokens` is answered by the router from the model's chat
template, without calling an engine. Image tokens are not counted, and a model
without a chat template answers 500 rather than a raw-text estimate.
`thinking` sets the `thinking` / `enable_thinking` chat template kwargs. The
engine's own `/v1/messages` also adapts it per reasoning parser (for example
`reasoning_effort` for Hunyuan), which the router cannot see; on such models
use `output_config.effort` or `chat_template_kwargs` directly.
## DeepSeek V4 and V4.1
Native V4 and V4.1 rendering comes from `rust/sglang-processor`, shared with SGLang's Rust
server, and follows SGLang's serving path (`serving_chat.py`), not Dynamo's
OpenAI defaults: all declared tools are rendered with SGLang's schema
defaults, reasoning effort comes from `reasoning` / `reasoning_effort`, and the
official/preview effort profile is detected from the checkpoint's
`encoding/encoding_dsv4.py` or overridden by `dsv4_reasoning_effort_profile` in
`config.json`, as in SGLang. Reference prompts live in `tests/fixtures/deepseek/`
and are regenerated by `tests/scripts/generate_deepseek_parity.py`.
V4.1 Flash uses Dynamo's V4.1 encoder with SGLang's numeric reasoning budgets
and tool payloads. Developer messages and media are left to the worker, since
the encoder renders them differently.
## GLM-5.3
GLM-5.3 and GLM-5.3-Flash render with the checkpoint's `chat_template.jinja` through
`rust/sglang-processor`, which builds the template context SGLang's Jinja path builds,
so every text chat forwards `input_ids`. Media, a `chat_template` kwarg, `-0` in tools
or tool arguments, and U+001C..U+001F in assistant history render on the worker. With
the cookbook's parsers, chats without tools are served through `/generate`; chats with
tools keep the engine's route (see above).
## Qwen3.8
Qwen3.8 (Max, 27B and Flash-Next) renders with the checkpoint's `chat_template.jinja`
through `rust/sglang-processor`, which picks it by its template and builds the
template context SGLang's Jinja path builds. Every text chat forwards `input_ids`.
Media, a `chat_template` key in `chat_template_kwargs`, and text or tool numbers
Python's `trim` or `json.dumps` treat unlike Dynamo's (such as `1e-06`) render on the
worker. With the cookbook's `qwen3` and `qwen3_coder` parsers, text chats are served
through `/generate`; thinking follows the template's `enable_thinking` (on by default).
## Kimi-K3
Kimi-K3 renders through `rust/sglang-processor` (dynamo-render's XTML formatter
with SGLang's request semantics) and is tokenized with the checkpoint's chunked
tiktoken encoding; shapes the formatter renders differently go to the worker.
`--tokenizer-path` accepts a
local `tiktoken.model` or an HF repo id, whose `tiktoken.model`, `config.json`
and `tokenizer_config.json` are downloaded when it has no `tokenizer.json`.
With the cookbook's `kimi_k3` parsers, text chats are served through `/generate`.
## HTTP/2
There is nothing to configure. The router negotiates per connection inbound and
resolves the protocol per worker outbound; every combination below is reached
automatically, and HTTP/1.1 remains a supported peer on both edges.
**Inbound.** The listener accepts cleartext HTTP/2 (h2c, prior knowledge) and
HTTP/1.1 on the same `--port`, chosen per connection. A mesh sidecar or
load balancer that prefers h2c multiplexes over one connection instead of
opening one per request; an HTTP/1.1 client is unaffected.
**Outbound.** At registration the router reads each worker's `/server_info` and
forwards over cleartext h2c only when that worker reports `--enable-http2` (the
engine's Granian server, which serves h2c and HTTP/1.1 together) **and** the
worker URL is cleartext. Everything else uses the default client, which speaks
HTTP/1.1 in cleartext and negotiates ALPN `h2, http/1.1` over TLS:
| `/server_info` | worker URL | router forwards over |
|---|---|---|
| `enable_http2: true` | `http://` | cleartext h2c |
| `enable_http2: true` | `https://` | HTTP/2 over TLS, by ALPN |
| `enable_http2: false`, or absent | any | HTTP/1.1 (cleartext) or ALPN (TLS) |
The choice is per worker, not fleet-wide, so a mixed fleet works and one
worker's state never changes another's. The flag comes from the `/server_info`
launch record, which has reported it all along, so no engine change is needed.
A worker whose `/server_info` does not answer has no readable protocol and
forwards over HTTP/1.1 for as long as it stays registered — a throughput cost,
never a correctness one. Admin fan-out (`/flush_cache`) always uses the default
client, because it addresses every worker at once rather than a selected one.
Two things worth knowing when debugging. h2c is prior-knowledge only — there is
no negotiation and no fallback — which is why the router requires the engine's
own `enable_http2` report before using it. And a worker's protocol is fixed
for as long as that worker stays registered: it is read once, at registration,
and never re-read. Under the K8s backend a restarting engine flips its
EndpointSlice to `ready=false`, which is a `Removed` → `Added` cycle and so a
fresh reading; under `--worker-urls` the fan-out happens once at startup and
nothing re-registers. So a worker registered over h2c that later stops serving
it (a proxy interposed on its port, say) is not detected until it
is re-registered; its circuit breaker will open in the meantime.
## Native gRPC
`--grpc-port` serves the engine's own gRPC API, `sglang.runtime.v1.SglangService`
(`proto/sglang/runtime/v1/sglang.proto`), so a gRPC client can point at the
router or at an engine. A call is routed exactly like its HTTP route and sent to
the same RPC on the selected engine's native gRPC server, which engines run
beside HTTP with their own `--grpc-port`; the router reads that port from each
worker's `/server_info`. HTTP requests are unaffected and still reach engines
over HTTP.
| RPC | Routed like |
|---|---|
| `ChatComplete` | `/v1/chat/completions`: same rendering, `input_ids` forwarding, PD and DP-rank handling |
| `Complete` | `/v1/completions` |
| `Generate`, `TextGenerate` | `/generate` with `input_ids` or `text` |
| `Embed`, `TextEmbed`, `OpenAIEmbed` | `/v1/embeddings` |
| `Classify`, `OpenAIClassify` | `/v1/classify` |
| `Rerank` | `/v1/rerank` |
The OpenAI RPCs carry the HTTP route's JSON body, so they are prepared exactly as
over HTTP. A typed RPC is prepared from its fields with the same rules, and the
router's additions (rid, PD bootstrap, DP rank, sampling defaults) are set on the
proto it forwards. A text prompt is tokenized once, as its HTTP route forwards
`input_ids`: `TextGenerate` is sent as `Generate` with `return_text`, so the engine
still supplies the text, and `TextEmbed` as `Embed`. They stay text RPCs under
`--disable-input-ids-forwarding`, without a tokenizer, or, for `TextGenerate`, with
`return_text_in_logprobs`. Other RPCs, such as
`Tokenize`, `HealthCheck` and the admin RPCs, answer `UNIMPLEMENTED`. Request metadata and `trace_headers`
act as HTTP headers for routing and forwarding, and a cancelled call aborts the
engine request. Router errors arrive as a gRPC status carrying the HTTP path's
message and an `x-router-error-code` trailer; an engine's status passes through
unchanged. An engine without `--grpc-port` (including one launched with
`--api-key`, which native gRPC rejects) cannot serve gRPC calls: the router warns
when it registers, and calls routed to it fail with `UNAVAILABLE`.
## Upgrading from `cache_aware_zmq`
The `cache_aware_zmq` policy has been removed. Configurations using it should
select `--policy cache_aware` and choose a native cache-prefix source: the
Router-local radix tree (the default), or the external Indexer shown above.
The legacy `--cache-threshold`, `--balance-abs-threshold`, and
`--balance-rel-threshold` flags have also been removed. They do not have
one-to-one replacements; remove them and review the current `sgl-router
--help` output when tuning Cache-Aware routing.
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

+1 -1
View File
@@ -100,7 +100,7 @@ pub struct ModelArgs {
/// Accepts temperature, top_p, top_k, min_p, repetition_penalty,
/// frequency_penalty, presence_penalty, and n. Numeric values fill absent
/// fields; {"min": LO, "max": HI} bands only constrain supplied values
/// and require reject mode. See README for domains and null handling.
/// and require reject mode. See REFERENCE.md for domains and null handling.
#[arg(long, value_name = "JSON")]
pub override_sampling_params: Option<String>,
@@ -1,7 +1,7 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 The SGLang Authors
// SPDX-License-Identifier: Apache-2.0
//! Side-by-side implementation of POLICY_DESIGN.md. Chat routing can opt into
//! Routing policies described in ARCHITECTURE.md. Chat routing can opt into
//! this interface through AppContext; `policies` remains the default.
pub mod admission;