[Doc] sgl-router: Rewrite README and add architecture guide (#43592)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Shangming Cai <csmthu@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
Shangming Cai
parent
e5dc10e73b
commit
62d4261cf4
@@ -0,0 +1,236 @@
|
||||
# SGLang router architecture
|
||||
|
||||
This document describes `--chat-routing reorg`: the deployment, request flow,
|
||||
and routing policies. The default legacy path is outside its scope.
|
||||
|
||||
- [1. Overview](#1-overview)
|
||||
- [2. Configuration examples](#2-configuration-examples)
|
||||
|
||||
## 1. Overview
|
||||
|
||||
### 1.1 Deployment Architecture
|
||||
|
||||
A deployment consists of a **router** and **buckets of SGLang engines**. A bucket
|
||||
is a logical group defined in the router configuration. It contains either a
|
||||
**unified engine group**, where each engine handles both prefill and decode,
|
||||
or separate **prefill and decode (P/D) groups**.
|
||||
|
||||
The example below shows short-context and long-context buckets, each with prefill
|
||||
and decode engines.
|
||||
|
||||

|
||||
|
||||
Each engine publishes load reports and KV events, which the router uses for
|
||||
routing. An optional [external KV indexer](sgl-kv-indexer/README.md) can supply
|
||||
prefix matches. The router also maintains session bindings and in-flight request
|
||||
counts.
|
||||
|
||||
To serve a request, the router chooses a bucket, selects engines within it, and
|
||||
forwards the request. In P/D, it sends the request to both the prefill and decode
|
||||
engines; the prefill engine transfers KV directly to the decode engine.
|
||||
|
||||
### 1.2 Request routing lifecycle
|
||||
Routing a request typically goes through **preprocessing** (rendering and
|
||||
tokenizing), **picking a bucket**, **scoring candidate engines** (load, prefix
|
||||
matching, or session affinity), **choosing engine(s)**, **forwarding** the
|
||||
request, and **handling output tokens** (e.g. tool parsing). Along the way it
|
||||
also applies **admission control**, **failure handling**, and **retries**.
|
||||
|
||||
### 1.3 Example: load-aware routing
|
||||
|
||||
The router tries each **bucket** that fits the request's length, ordered by SLO
|
||||
preference, capacity, then `rank`; a request that fits no bucket is rejected
|
||||
with **400** before any engine is considered. In each bucket it drops the
|
||||
engines that fail **admission**, then picks the **less loaded of two engines
|
||||
sampled** from those that remain; when no engine admits the request it moves to
|
||||
the next bucket, and returns **503** when no bucket has one. If **dispatch**
|
||||
fails before any response and **retries** remain, it excludes that engine and
|
||||
restarts from the first bucket.
|
||||
|
||||
**Load** is a vector of engine metrics compared field by field, and the first
|
||||
difference decides. For **unified and prefill** engines the order is estimated
|
||||
prefill queue time, waiting uncached tokens, waiting requests, running requests,
|
||||
then in-flight requests; **decode** engines rank by waiting and running requests,
|
||||
then KV usage.
|
||||
|
||||
```mermaid
|
||||
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#eaf3fb", "primaryTextColor": "#172b4d", "primaryBorderColor": "#345779", "lineColor": "#345779", "edgeLabelBackground": "#ffffff"}, "flowchart": {"curve": "stepAfter", "rankSpacing": 30, "nodeSpacing": 30}}}%%
|
||||
flowchart TD
|
||||
R([Request]) --> L{"0. Fits any bucket's<br/>length limits?"}
|
||||
L -->|no| Y(["400 to client"])
|
||||
L -->|yes| B["1. Take next compatible bucket"]
|
||||
B --> A{"2. Any engine can admit<br/>the request?"}
|
||||
A -->|yes| S["3. Sample two admitted engines<br/>Pick the less loaded one"]
|
||||
S --> D["4. Dispatch to engine"]
|
||||
A -->|no| M{"More buckets?"}
|
||||
M -->|no| X(["503 to client"])
|
||||
D -->|success, or<br/>out of retries| E([Response to client])
|
||||
D -->|failed before response,<br/>retries left| T["Exclude failed engine"]
|
||||
M -.->|yes: next bucket| B
|
||||
T -.->|restart at first bucket| B
|
||||
classDef next fill:#fff5e6,stroke:#ac762c,color:#654516;
|
||||
classDef done fill:#e8f5ef,stroke:#46826a,color:#234d3c;
|
||||
class T,X,Y next;
|
||||
class E done;
|
||||
```
|
||||
|
||||
### 1.4 Example: cache-aware and session-aware routing
|
||||
|
||||
These policies replace steps **2–3** above. `cache_aware` serves plain and
|
||||
prefill groups only; decode groups use load-aware (`power_of_two`) unless they set
|
||||
`session_aware`. Both policies split the group's engines in two:
|
||||
|
||||
- **Preferred engine (A):** for `cache_aware`, the best-ranked prefix owner that
|
||||
passes admission; for `session_aware`, the engine bound to the session.
|
||||
- **Normal engines:** the rest of the engine group. When needed, the router picks
|
||||
one of them (**B**) with the load-aware `power_of_two` policy.
|
||||
|
||||
A and B are checked against the same group admission limits (for example
|
||||
`max_kv_usage` or `max_waiting_requests`; see [configuration examples](#2-configuration-examples)).
|
||||
|
||||
- **`prefer` mode (default):** use A if it passes admission.
|
||||
- **`balanced` mode:** also pick a B engine and compare the loads of A and B.
|
||||
- Switch to B only when it is clearly less loaded:
|
||||
`load(A) > load_factor × load(B)` **and** `load(A) − load(B) > load_gap`
|
||||
(defaults: factor 2; gap 1024 tokens, or 4 requests). Otherwise, including
|
||||
when either load is unknown, keep A.
|
||||
- `balanced_by` sets the load metric:
|
||||
- **`prefill_tokens` (default):** the engine's reported waiting uncached tokens,
|
||||
plus the tokens this request would prefill on it. For `cache_aware` that
|
||||
subtracts the engine's cached prefix, so A's cache hit counts in its favor.
|
||||
- **`running_requests`:** the engine's reported running requests.
|
||||
- **No A, or A fails admission (either mode):** fall back to B.
|
||||
|
||||
## 2. Configuration examples
|
||||
|
||||
**CLI flags** set the defaults for every engine group. A **bucket file**
|
||||
(`--bucket-config`) defines the buckets, and each group in it can override the
|
||||
policy, admission, and affinity settings for itself. Without a bucket file,
|
||||
there are no length tiers: every request can go to any engine.
|
||||
|
||||
Settings fall into these categories:
|
||||
|
||||
| Category | What it controls | CLI flags | Bucket-file group key |
|
||||
| --- | --- | --- | --- |
|
||||
| **Engines** | Plain or P/D, and where engines come from | `--worker-urls`, or `--service-discovery` with `--selector` (plain) or `--prefill-selector` + `--decode-selector` (P/D) | `worker_services`, `worker_ids` |
|
||||
| **Policy** | How the router scores engines within a group | `--policy` | `policy` |
|
||||
| **Affinity** | When the preferred engine (prefix owner or session binding) wins over a normal engine | `--affinity-*`, `--cache-affinity-*`, `--session-id-header` | `affinity` |
|
||||
| **Admission** | Limits that make an engine reject a request | `--max-*` | `admission` |
|
||||
| **Buckets** | Which buckets a request fits and the order they are tried | `--bucket-config` | `max_input_tokens`, `max_context_tokens`, `rank`, ... |
|
||||
|
||||
How policies apply to each group:
|
||||
|
||||
- **Plain and prefill groups** use `--policy` (default `power_of_two`).
|
||||
- **Decode groups** always default to `power_of_two`; under reorg,
|
||||
`--decode-policy` must stay `power_of_two`.
|
||||
- **A bucket-file group** can set `"policy"` to `power_of_two` or to the
|
||||
`--policy` value, nothing else. `cache_aware` cannot be used on decode, so
|
||||
only `session_aware` can replace `power_of_two` there.
|
||||
|
||||
### Example A: plain engines, cache-aware
|
||||
|
||||
```bash
|
||||
sgl-router --model-id qwen3 --chat-routing reorg \
|
||||
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
|
||||
--policy cache_aware \
|
||||
--affinity-mode prefer --cache-affinity-min-match-ratio 0.2 \
|
||||
--max-waiting-requests 16 --max-in-flight 56
|
||||
```
|
||||
|
||||
- **Engines:** two plain engines; each handles both prefill and decode.
|
||||
- **Policy:** every engine is scored with `cache_aware`.
|
||||
- **Affinity:**
|
||||
- An engine owns the prefix if it holds at least 1024 matched tokens (the
|
||||
default) **and** 20% of the prompt.
|
||||
- `prefer` sends the request to the best owner that passes admission.
|
||||
- **Admission:** an engine is rejected when it reports 16 or more waiting
|
||||
requests, or has 56 or more requests in flight from this router.
|
||||
|
||||
### Example B: P/D engines, session-aware
|
||||
|
||||
```bash
|
||||
sgl-router --model-id qwen3 --chat-routing reorg \
|
||||
--service-discovery --service-discovery-namespace inference \
|
||||
--prefill-selector app=qwen3,role=prefill \
|
||||
--decode-selector app=qwen3,role=decode \
|
||||
--policy session_aware --session-id-header x-session-id \
|
||||
--affinity-mode balanced --affinity-balanced-by running-requests \
|
||||
--max-kv-usage 0.9
|
||||
```
|
||||
|
||||
- **Engines:** Kubernetes discovery finds prefill and decode engines by label.
|
||||
- **Policy:**
|
||||
- Prefill engines use `session_aware`: requests with the same
|
||||
`x-session-id` stick to the prefill engine they are bound to.
|
||||
- Decode engines use `power_of_two`. To make decode session-aware too, use a
|
||||
bucket file whose decode group sets `"policy": "session_aware"`.
|
||||
- **Affinity:** in balanced mode the router samples an alternative engine, and
|
||||
the session moves to it when the bound engine has more than 2x the
|
||||
alternative's running requests **and** more than 4 more than it (the
|
||||
defaults). The session also moves when the bound engine fails admission.
|
||||
- **Admission:** prefill and decode engines are rejected at 90% KV usage.
|
||||
|
||||
### Example C: P/D buckets from a file
|
||||
|
||||
```bash
|
||||
sgl-router --model-id qwen3 --chat-routing reorg \
|
||||
--service-discovery --service-discovery-namespace inference \
|
||||
--prefill-selector app=qwen3,role=prefill \
|
||||
--decode-selector app=qwen3,role=decode \
|
||||
--policy cache_aware \
|
||||
--bucket-config buckets.json
|
||||
```
|
||||
|
||||
`buckets.json` defines a short tier with a TTFT estimate and a long tier:
|
||||
|
||||
```json
|
||||
{
|
||||
"ttft_slo": "slo_first",
|
||||
"buckets": [
|
||||
{
|
||||
"id": "short",
|
||||
"max_input_tokens": 4096,
|
||||
"max_context_tokens": 8192,
|
||||
"ttft_ms": 300,
|
||||
"prefill": {
|
||||
"worker_services": ["inference/prefill-short"],
|
||||
"admission": {"max_pending_prefill_tokens": 32768},
|
||||
"affinity": {"mode": "balanced", "load_factor": 1.5}
|
||||
},
|
||||
"decode": {
|
||||
"worker_services": ["inference/decode-short"],
|
||||
"admission": {"max_kv_usage": 0.9}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "long",
|
||||
"rank": 1,
|
||||
"max_context_tokens": 131072,
|
||||
"prefill": {
|
||||
"worker_services": ["inference/prefill-long"],
|
||||
"policy": "power_of_two"
|
||||
},
|
||||
"decode": {"worker_services": ["inference/decode-long"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- **Engines:** each group takes the discovered engines of its role that belong
|
||||
to the listed Kubernetes Services.
|
||||
- **Buckets:**
|
||||
- Requests that fit within 4k input and 8k context fit both buckets; `short`
|
||||
is tried first because it is smaller. Longer requests go only to `long`.
|
||||
- If `short` rejects a request, the router tries `long`.
|
||||
- **Policy:**
|
||||
- `short` prefill uses `cache_aware` from `--policy`.
|
||||
- `long` prefill overrides it with `power_of_two`.
|
||||
- Both decode groups use `power_of_two`.
|
||||
- **Affinity:** `short` prefill uses balanced mode with a load factor of 1.5.
|
||||
`power_of_two` groups take no `affinity` block; setting one is rejected at
|
||||
startup.
|
||||
- **Admission:**
|
||||
- `short` prefill engines reject requests while 32k or more uncached tokens
|
||||
are waiting.
|
||||
- `short` decode engines reject requests at 90% KV usage.
|
||||
- `long` engines set no limits.
|
||||
@@ -1,655 +0,0 @@
|
||||
# Engine selection: target design
|
||||
|
||||
This document describes the target architecture for engine selection. It defines
|
||||
responsibilities and behavior; the interface and configuration examples are
|
||||
sketches, not a specification of the current API or CLI. Implementation status is
|
||||
listed at the end.
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Filter buckets by token length, then order preferences.** `BucketResolver`
|
||||
returns all compatible buckets, applying optional SLO preferences before
|
||||
capacity/rank/ID ordering, without inspecting workers or policies.
|
||||
2. **The bucket owns plain versus PD engine selection.** `Bucket::pick_engines`
|
||||
calls its one plain group or both prefill and decode groups, returning a
|
||||
complete selection. Both PD engines come from that same bucket.
|
||||
3. **An engine group owns membership and its policy.** `EngineGroup::pick`
|
||||
filters live workers by model, health, stage, and membership, invokes the
|
||||
policy, and validates that its result belongs to the exact candidate set.
|
||||
4. **A policy owns selection, fallback, admission, and its state dependencies.**
|
||||
Construction injects the load, KV, or affinity handles it needs. Request-time
|
||||
arguments contain facts and candidates. A policy cannot choose another bucket
|
||||
or cross a PD role boundary.
|
||||
5. **The handler owns bucket fallback and dispatch.** It calls `pick_engines`
|
||||
on each bucket and dispatches only after a complete selection succeeds.
|
||||
Missing candidates or admission rejection advance to the next bucket, where
|
||||
all required engines are selected again. Invalid signals or policy results stop routing.
|
||||
|
||||
An engine is represented by `Worker`. Registry role labels remain authoritative
|
||||
when filtering candidates. Groups reference worker IDs rather than owning live
|
||||
workers. Group policy instances are reused across requests.
|
||||
|
||||
### Data ownership
|
||||
|
||||
| Type | Owns |
|
||||
| --- | --- |
|
||||
| `BucketResolver` | A model's bucket collection, length filtering, and ordering |
|
||||
| `Bucket` | ID, length constraints, rank, groups, and complete plain/PD engine selection |
|
||||
| `EngineGroup` | Engine membership, attached policy, and engine selection |
|
||||
| `WorkerRegistry` | Live workers, model membership, health, and role |
|
||||
| Request handler | Request preparation, ordered bucket attempts, HTTP errors, and dispatch |
|
||||
|
||||
Omitting both `worker_ids` and `worker_services` includes every healthy engine
|
||||
serving the requested model and role. Explicit empty membership lists are rejected
|
||||
by the config factory. `EngineGroup::new(policy)` creates a catch-all group.
|
||||
`BucketGroups::Pd` requires both groups, making partial or mixed plain/PD bucket
|
||||
configurations unrepresentable.
|
||||
|
||||
## 1. Code organization
|
||||
|
||||
```text
|
||||
src/
|
||||
buckets.rs Bucket resolution and engine-group selection
|
||||
policies/
|
||||
mod.rs Policy contract and construction
|
||||
admission.rs Per-engine acceptance checks
|
||||
cache_aware.rs Cache selection and local/remote adapter
|
||||
session_aware.rs Session selection and fallback
|
||||
sticky.rs Routing-key selection and fallback
|
||||
least_load.rs Least-load selection
|
||||
power_of_two.rs Pair sampling and stage-aware comparison
|
||||
random.rs Random selection
|
||||
round_robin.rs Rotation
|
||||
state/
|
||||
mod.rs Shared exports
|
||||
kv_events/ Local index, subscriptions, hashing, wire format
|
||||
load_monitor/ Engine reports and router-local request accounting
|
||||
affinity_store.rs Assignments, expiry, and atomic updates
|
||||
server/
|
||||
app_context.rs Shared service lifecycle and policy wiring
|
||||
routes/chat.rs Request preparation, PD coordination, dispatch
|
||||
```
|
||||
|
||||
During migration, `buckets_reorg.rs` and `policies_reorg/` implement this design
|
||||
beside the live `policies/` path. The layout above is the target after switchover.
|
||||
Shared state already lives directly under `src/state/`.
|
||||
|
||||
Dependencies flow from engine groups to policies, and from policies and admission
|
||||
to shared state. State does not depend on bucket ordering or concrete policy
|
||||
strategies. Small shared helpers are sufficient; no generic score-composition,
|
||||
tier executor, or separate selection framework is required.
|
||||
|
||||
## 2. Responsibilities and request flow
|
||||
|
||||
`BucketResolver::resolve(input_tokens, expected_peak_tokens, ttft_ms, tokens_per_second)` returns an ordered
|
||||
list of compatible bucket references (possibly empty), or an invalid-signal error.
|
||||
It does not receive a stage or a load view, resolve live engines, or invoke policies.
|
||||
The handler iterates this list until a bucket supplies the complete engine selection.
|
||||
|
||||
`Bucket::pick_engines(workers, request)` accepts a `BucketRequest`
|
||||
of prepared routing facts, invokes the required groups, and returns `BucketPick`
|
||||
(one plain pick or a complete P/D pair). Failures retain their stage. This API
|
||||
has no HTTP headers, `AppContext`, or forwarding dependency.
|
||||
|
||||
`EngineGroup::pick(workers, request)` resolves healthy workers for the request's
|
||||
model and stage, intersects them with its membership, sorts them by stable ID,
|
||||
and invokes its attached policy. It rejects foreign results, including a newly
|
||||
allocated worker with the same ID as a candidate.
|
||||
|
||||
```text
|
||||
chat_completions (reorg configured): prepare tokens and expected peak
|
||||
|
|
||||
v
|
||||
BucketResolver::resolve: ordered length-compatible buckets
|
||||
|
|
||||
v
|
||||
For each bucket: bucket.pick_engines(...)
|
||||
|
|
||||
+-- BucketGroups::Plain
|
||||
| plain.pick() -> one plain engine
|
||||
|
|
||||
+-- BucketGroups::Pd
|
||||
prefill.pick() -> P engine
|
||||
decode.pick() -> D engine from the same bucket
|
||||
|
|
||||
+-- empty group / admission rejection -> try next bucket (repeat all picks)
|
||||
+-- invalid signal / configuration / foreign pick -> return error
|
||||
|
|
||||
v
|
||||
Complete selection -> forward_chat_request: acquire guards, attach PD bootstrap, forward response
|
||||
```
|
||||
|
||||
The handler extracts token facts and header keys once into `BucketRequest`.
|
||||
The bucket creates a stage-specific `PickRequest` for each group call, supplying
|
||||
its own ID and the role associated with that group. Input length, expected peak,
|
||||
token IDs, and session/routing keys pass through. Policies obtain observations
|
||||
from their own shared-state handles; buckets and handlers do not provide load,
|
||||
KV, or affinity services on each call. PD uses separate group policies, but never
|
||||
independently resolves a decode bucket. Decode selection
|
||||
failure discards that tentative prefill choice and advances to the next bucket
|
||||
on missing candidates or admission rejection. No forwarding guards are acquired
|
||||
and no prefill request is sent until both picks in one bucket succeed.
|
||||
PD compatibility constraints beyond model and role remain follow-up work.
|
||||
|
||||
There is one endpoint: `POST /v1/chat/completions`. `AppContext::chat_routing`
|
||||
chooses its implementation:
|
||||
|
||||
- `ChatRouting::Legacy` (default) uses the existing policies and bucket selector.
|
||||
- `ChatRouting::Reorg(HashMap<ModelId, BucketResolver>)` uses the new bucket and
|
||||
policy interfaces, with explicit model-specific resolvers.
|
||||
|
||||
`--chat-routing reorg` constructs default plain/PD buckets using the existing
|
||||
`--policy` and tuning flags. Programmatic callers can still install explicit
|
||||
resolvers. A missing model returns 404 without falling back to legacy routing.
|
||||
The default serving path remains legacy.
|
||||
|
||||
Both implementations reuse request preparation (including sampling validation
|
||||
and tokenization), forwarding, streaming, middleware, and the 32 MiB body limit.
|
||||
The reorg implementation requests tokenization for length matching, retaining
|
||||
the existing body-size estimate when tokenization is unavailable.
|
||||
|
||||
## 3. Bucket resolution
|
||||
|
||||
1. Validate that a known expected peak is at least the input length.
|
||||
2. Keep buckets whose inclusive input-token range contains the input length.
|
||||
3. Check the bucket context capacity against input plus requested output when
|
||||
known, or against input length when the output budget is unknown.
|
||||
4. Apply enabled SLO preferences to complete buckets. Count unmet preferences
|
||||
equally, then sort by ascending input capacity (the lesser of the input upper
|
||||
bound and context capacity), rank, and ID. Unbounded capacities sort last.
|
||||
Return the entire ordered list, retaining nonpreferred buckets for fallback.
|
||||
5. The handler calls each bucket's `pick_engines` until one supplies its complete
|
||||
selection. A failed PD attempt never contributes an engine to a later pair.
|
||||
|
||||
The handler checks addition overflow when computing the expected peak.
|
||||
An empty bucket list becomes a 400 `NoMatchingBucket` response. After exhausting
|
||||
the list, accumulated admission rejection details produce a selection failure
|
||||
(503); if there were no admission rejections, the last unavailable stage produces
|
||||
a stage-specific 503. Invalid policy signals/configuration or out-of-candidate
|
||||
picks stop the pass immediately with an internal error. Successful engine
|
||||
selection ends the pass; forwarding errors do not restart bucket iteration.
|
||||
|
||||
Token ranges and rank belong to the bucket, not its engine groups. For a PD
|
||||
bucket, both groups share this one request-length decision. Policy fallback on
|
||||
a cache/affinity miss stays within that group's candidates. There is no second
|
||||
pass with relaxed admission and no post-policy substitution.
|
||||
|
||||
### Optional SLO ordering
|
||||
|
||||
`Bucket` has optional `ttft_ms` and `tokens_per_second` estimates. The resolver
|
||||
has independent `ttft_slo` and `tps_slo` preferences: `Disabled` (default),
|
||||
`SloFirst` (matching first), and `BestEffort` (nonmatching first). The handler
|
||||
parses `x-sgl-ttft-slo-ms` and `x-sgl-tps-slo` only when their preference is enabled;
|
||||
invalid enabled headers return 400 before dispatch. Disabled headers are ignored.
|
||||
SLO targets belong to bucket resolution, not the engine policy's `PickRequest`.
|
||||
|
||||
Absent targets are neutral. A bucket matches TTFT when its positive estimate is
|
||||
at most the target, and throughput when its finite positive estimate is at least
|
||||
the target. Missing or invalid estimates do not match a supplied target. Enabled
|
||||
TTFT targets must be positive; throughput targets must be finite and positive.
|
||||
|
||||
Each unmet preference adds one ordering penalty. With both preferences set to
|
||||
`SloFirst`, a bucket matching both comes before one matching either, followed by
|
||||
buckets matching neither. Capacity/rank/ID breaks ties within these tiers. The
|
||||
same logic applies to plain and PD buckets, and a PD bucket always supplies both
|
||||
engines. TTFT and throughput preferences never independently resolve P/D groups.
|
||||
Length constraints are applied first and admission rejection still advances to
|
||||
the next complete bucket, including a bucket outside the preferred SLO tier.
|
||||
|
||||
Global session modes and sticky policies are follow-ups. Their
|
||||
integration must preserve bucket-first selection and the same-bucket PD rule.
|
||||
Cross-bucket affinity probing is not part of this interface. Session/routing
|
||||
keys still pass through `PickRequest` for policies operating inside the selected
|
||||
group; unsupported legacy modes need explicit migration decisions before the
|
||||
standard serving path switches.
|
||||
|
||||
## 4. Policy and admission contracts
|
||||
|
||||
### Policy
|
||||
|
||||
All policies implement one asynchronous, object-safe `Policy::pick` interface.
|
||||
A boxed future supports the remote prefix indexer; local policies can return an
|
||||
immediately ready result.
|
||||
|
||||
```rust
|
||||
pub trait Policy: Send + Sync + std::fmt::Debug {
|
||||
fn pick<'a>(
|
||||
&'a self,
|
||||
engines: &'a [Arc<Worker>],
|
||||
request: &'a PickRequest<'a>,
|
||||
) -> BoxFuture<'a, Result<Pick, PickError>>;
|
||||
}
|
||||
```
|
||||
|
||||
Policies own their required state handles and read observations locally. Nested
|
||||
fallback uses `pick_fallback(engines, request)`, which calls the fallback's `pick`;
|
||||
the fallback reads its own state. There is no shared observation context or cache.
|
||||
Buckets and HTTP handlers supply only candidates and request facts.
|
||||
|
||||
`Pick` identifies one engine and a selection reason for metrics and tracing.
|
||||
`PickRequest` carries model, stage, selected bucket ID, input and optional
|
||||
expected peak counts, optional token IDs, and session/routing keys. It contains
|
||||
no HTTP body, bucket resolver, state handles, snapshots, or backend configuration.
|
||||
|
||||
| Outcome | Meaning |
|
||||
| --- | --- |
|
||||
| `Pick` | The chosen engine belongs to the supplied set and passed admission |
|
||||
| `NoMatchingBucket` | No bucket supports the requested length |
|
||||
| `NoCandidates` | No eligible member or policy selection miss |
|
||||
| `NoAdmissibleEngine` | A policy exhausted its candidates; includes rejection reasons |
|
||||
| `AdmissionRejected` | The chosen engine failed after-selection admission |
|
||||
| `InvalidSignal` / `InvalidConfiguration` | Invalid policy input or configuration |
|
||||
| `OutsideCandidates` | Policy returned an engine outside its exact candidate set |
|
||||
|
||||
### Admission
|
||||
|
||||
Admission evaluates acceptance. It does not rank engines, choose replacements,
|
||||
change buckets, or mutate affinity.
|
||||
|
||||
An admission policy is a set of per-engine caps, `AdmissionLimits`. Each cap
|
||||
is optional; an unset cap is not checked, and the default admits everything.
|
||||
Each cap admits while the engine's load is below it. Usage caps are shares, in
|
||||
(0, 1], of the engine's reported capacity, so one limit fits engines of any
|
||||
size. The request itself is not counted: an engine just below a cap still takes
|
||||
it, and the engine queues or retracts if it overflows. Admission observes load
|
||||
without reserving it.
|
||||
|
||||
| Limit | Checks |
|
||||
| --- | --- |
|
||||
| `max_running_usage` | Running requests against reported max running requests |
|
||||
| `max_kv_usage` | KV tokens against reported KV capacity |
|
||||
| `max_waiting_requests` | Reported waiting requests |
|
||||
| `max_pending_prefill_tokens` | Reported waiting uncached tokens |
|
||||
| `max_inflight_requests` | Router-local in-flight requests |
|
||||
|
||||
```json
|
||||
{"max_running_usage": 0.9, "max_kv_usage": 0.95, "max_inflight_requests": 64}
|
||||
```
|
||||
|
||||
Unknown fields are rejected during deserialization.
|
||||
|
||||
The policy reads the selected engine's `EngineMetrics` from the load snapshot
|
||||
it already captured for selection plus the live in-flight counter and calls
|
||||
`EngineAdmission::check(engine, metrics)`, which returns `Allow`,
|
||||
`Reject(limit name)`, or an error. Reported
|
||||
metrics are `None` without a fresh, complete report, never zero, and such
|
||||
limits fail open; the in-flight count is always known. Policies attach the
|
||||
checker as `Arc<dyn EngineAdmission>` and decide where checking belongs in
|
||||
their selection algorithm; there is no placement setting or filtering wrapper.
|
||||
|
||||
Power-of-two first selects an engine, then calls admission exactly once on that
|
||||
engine. A rejection returns `AdmissionRejected` to the bucket loop; it does not
|
||||
resample, choose the other sampled engine, or run a policy fallback. No candidates
|
||||
returns `NoCandidates` without invoking admission. A single candidate is selected
|
||||
directly; otherwise two distinct candidates are sampled uniformly, and the one
|
||||
with lower stage pressure wins. A complete tie keeps the first sampled engine.
|
||||
|
||||
Power-of-two reuses the existing pure pressure-comparison functions. Plain and
|
||||
prefill stages compare estimated prefill queue time when both reports provide it,
|
||||
then waiting uncached tokens, waiting requests, and running requests. Decode
|
||||
compares waiting requests, running requests, KV usage fraction, then used KV tokens.
|
||||
Reported-pressure ties use router-local active requests. If either sampled engine
|
||||
lacks a fresh, complete native report with valid capacity, both are compared by
|
||||
router-local active requests instead. Basic reports from older publishers are
|
||||
still passed to admission when fresh, but do not supply native pressure metrics.
|
||||
|
||||
Power-of-two retains the selected engine's load record from selection and
|
||||
passes it to admission without another snapshot. A single candidate still has
|
||||
its load read for admission, even though selection needs no comparison.
|
||||
Neither the bucket nor HTTP handler supplies observations. Synchronous checks
|
||||
do not fetch telemetry over the network themselves.
|
||||
|
||||
`AdmissionLimits::default()` is the default for new explicit policy attachments.
|
||||
It leaves health, role, membership, and policy preferences in force. Migrated
|
||||
configurations must retain their existing capacity and configured budget checks;
|
||||
see compatibility below.
|
||||
|
||||
Affinity preferences never bypass attached hard admission.
|
||||
|
||||
Admission checks observe capacity; they do not reserve it. Concurrent requests
|
||||
may pass against the same observation. Strict reservations would require a
|
||||
separate mechanism.
|
||||
|
||||
## 5. Concrete policies
|
||||
|
||||
| Policy | Selection behavior |
|
||||
| --- | --- |
|
||||
| `RoundRobinPolicy` | Rotate over candidates using a cursor owned by this policy instance |
|
||||
| `RandomPolicy` | Choose uniformly from candidates |
|
||||
| `PowerOfTwoPolicy` | Sample two distinct candidates when possible and choose the lower-pressure engine using the stage's load comparison |
|
||||
| `LeastLoadPolicy` (`load_based`) | Choose the least loaded engine; preserve tie-breaking, telemetry fallback, and recent-dispatch correction |
|
||||
| `SessionAwarePolicy` | Prefer the session binding; use admitted power-of-two fallback and rebind |
|
||||
| `StickyPolicy` | Reuse an admitted routing-key binding; use the configured fallback for new or missing keys |
|
||||
| `CacheAwarePolicy` | Prefer an admitted prefix owner; use power-of-two fallback |
|
||||
|
||||
Session assignments are scoped by model, bucket ID, stage, and session key.
|
||||
`SessionAwarePolicy::new(store, engine_load)` receives shared state; the caller
|
||||
owns the store's idle timeout and eviction task. Missing or empty session keys
|
||||
use power-of-two without creating assignments. A new or out-of-group binding
|
||||
uses power-of-two with `AdmissionLimits::default()`, then the session policy
|
||||
checks its selected engine before binding. A concurrent live assignment wins,
|
||||
but is checked before returning it; rejection ends that attempt without
|
||||
rewriting the binding or retrying another engine. Existing bindings follow the
|
||||
shared affinity modes below. Session policies can be attached
|
||||
independently to each role. Programmatic reorg callers configure
|
||||
`model.affinity.session_id_header` for HTTP header extraction; this does not
|
||||
enable legacy global modes or backup escape.
|
||||
|
||||
Session and sticky policies do not create assignments for missing keys. A
|
||||
binding outside the candidates cannot win. A missing binding may invoke policy
|
||||
fallback within the group. Rejected session affinity tries fallback without the
|
||||
old engine; a successful admitted replacement rebinds the session. Failed fallback
|
||||
preserves the old binding and advances to the next bucket.
|
||||
|
||||
Both affinity policies support `--affinity-mode prefer` (default) and `balanced`.
|
||||
Prefer retains admissible affinity. Balanced compares it with an admitted
|
||||
power-of-two alternative and switches only when the `--affinity-balanced-by` load
|
||||
exceeds both the alternative times `--affinity-load-factor` (default 2) and the
|
||||
alternative plus `--affinity-load-gap`. The load is `prefill-tokens` (default;
|
||||
gap default 1024): native waiting uncached tokens plus the request's uncached
|
||||
tokens on that engine, so the alternative also pays for the prefix the cache owner
|
||||
holds. Session-aware has no prefix signal and counts the whole input on both
|
||||
engines. `running-requests` (native or basic reports, gap default 4) ignores the
|
||||
request. Both reports must be fresh; missing data
|
||||
or ties preserve affinity. Bindings commit during selection after
|
||||
admission, as with initial placement; dispatch failure does not roll them back.
|
||||
|
||||
Sticky fallback supports `round_robin`, `random`, `power_of_two`, and `load_based`,
|
||||
with round-robin as the default. Nested fallbacks use
|
||||
`AdmissionLimits::default()`; the owning policy explicitly checks the engine
|
||||
returned by its fallback.
|
||||
|
||||
### Cache-aware behavior
|
||||
|
||||
The architecture preserves the cache algorithm within the supplied candidate
|
||||
set. Its responsibilities are:
|
||||
|
||||
1. Look up prefix ownership through the local index or remote indexer.
|
||||
2. Apply minimum matched-token and optional ratio thresholds.
|
||||
3. Bound candidates using prefix/pressure ordering and the configured minimum,
|
||||
ratio, and maximum worker counts.
|
||||
4. Check admission and choose the best remaining prefix holder.
|
||||
5. Apply the shared affinity mode. With no admitted prefix owner, fall back
|
||||
within the group, excluding rejected engines.
|
||||
|
||||
Fallback samples two engines and checks the selected engine's admission; it
|
||||
does not resample on rejection. Candidate limits apply only to prefix selection.
|
||||
|
||||
Memoize the prefix lookup once per request, including remote I/O. Each policy
|
||||
restricts those matches to its own candidates. A memoized lookup does not imply
|
||||
that another bucket or stage's cache selection and admission have already run.
|
||||
Any optimization that skips those steps must establish that the previous result
|
||||
applies to the current group.
|
||||
|
||||
## 6. Shared state and construction
|
||||
|
||||
Application wiring starts shared services once. Policy construction validates
|
||||
configuration and passes the required handles to each policy and admission
|
||||
implementation. For example, `PowerOfTwoPolicy::new(Arc<EngineReportedLoadTable>)`
|
||||
retains the application's shared load table. KV-aware and affinity-aware policies
|
||||
receive their corresponding shared handles when implemented. Policies with no
|
||||
state dependency require none. Policy instances do not create duplicate
|
||||
subscriptions, polling loops, indexes, or remote-client concurrency limits.
|
||||
|
||||
Requirements come from all configured policies, their admission checks, and
|
||||
nested fallbacks. This includes tokenization, affinity-header extraction, load
|
||||
observations, and dispatch timestamps. A role-group override must receive its
|
||||
required settings even when the model's default uses a different policy.
|
||||
|
||||
### Load state
|
||||
|
||||
`state/load_monitor/` owns engine reports and existing router-local request
|
||||
accounting. Power-of-two owns an `Arc<EngineReportedLoadTable>` and captures a snapshot
|
||||
locally for each nonempty selection attempt. Its `pick` method selects the engine,
|
||||
then passes that engine's borrowed load record directly to admission.
|
||||
No snapshot or observation is added to `Pick`, `PickRequest`,
|
||||
or the bucket interface, and no shared observation context is threaded through
|
||||
policies or fallbacks.
|
||||
|
||||
The existing snapshot reader preserves rank aggregation, freshness, and capacity
|
||||
fields. Missing, stale, or rank-incomplete reports yield `None`, not zero load.
|
||||
A new pick reads current state. Admission reuses the selected observation even if
|
||||
reports change after selection; it neither recaptures nor reserves capacity.
|
||||
Fallback policies read their own state and do not share snapshots with callers.
|
||||
|
||||
Snapshot capture still scans the full table; an engine-scoped reader can be added
|
||||
if profiling justifies it. Power-of-two ranks engines with the shared prefill/decode
|
||||
comparisons in `state/load_monitor/engine_ranking.rs`, including router-local
|
||||
fallback. Concrete load-aware admission remains
|
||||
in #40271. Further shared load interpretation and correction for dispatches since
|
||||
the report remain follow-ups; these must preserve source, freshness, and available
|
||||
measurements without adding another
|
||||
independent in-flight counter. Load and cache observations are not an atomic global
|
||||
snapshot. Preserve request-guard cleanup.
|
||||
|
||||
### Cache state
|
||||
|
||||
`state/kv_events/` owns local subscriptions, event application, hashing, and the
|
||||
radix-tree index. The tree stores prefix ownership and storage tiers, not KV
|
||||
tensors. Eviction, invalidation, and worker removal update this shared state.
|
||||
|
||||
A small `CacheSource` adapter beside the cache policy normalizes local and remote
|
||||
lookups into common prefix results. Preserve model/cache namespaces, required
|
||||
rank information, and hash/block-size handling. Convert block counts to token
|
||||
counts only with a valid conversion.
|
||||
|
||||
A confirmed miss and an unavailable backend both allow cache fallback, but
|
||||
remain distinguishable in diagnostics. Invalid queries and configuration errors
|
||||
remain explicit errors. Remote indexing and load-only deployments do not need
|
||||
a duplicate local cache tree.
|
||||
|
||||
### Affinity state
|
||||
|
||||
`AffinityStore` owns scoped assignments, idle expiry, worker invalidation, and
|
||||
atomic updates. Policies decide when to reuse or replace an assignment.
|
||||
|
||||
Concurrent first assignments must converge on an effective binding that is
|
||||
still a candidate and passes admission. If a concurrent binding returns a
|
||||
different engine, the policy must check that engine before returning it.
|
||||
Reconcile conflicts with bounded retry.
|
||||
|
||||
Create or replace a binding only after admission succeeds. A binding records
|
||||
preferred placement, not successful execution, so it may remain if later PD
|
||||
selection or dispatch fails. It must not increment dispatch accounting.
|
||||
|
||||
## 7. Configuration and compatibility
|
||||
|
||||
Under `--chat-routing reorg`, `--bucket-config` reads a JSON file of complete
|
||||
buckets. Each bucket sets `plain`, or both `prefill` and `decode`; each group may
|
||||
set `worker_ids` or `worker_services`, plus `policy`, `admission` and `affinity`.
|
||||
`affinity` overrides `mode`, `balanced_by`, `load_factor` and `load_gap` for a
|
||||
session- or cache-aware group; unset fields take the CLI values.
|
||||
|
||||
```json
|
||||
{
|
||||
"ttft_slo": "slo_first",
|
||||
"buckets": [
|
||||
{
|
||||
"id": "short-context",
|
||||
"max_input_tokens": 4096,
|
||||
"max_context_tokens": 8192,
|
||||
"ttft_ms": 300,
|
||||
"prefill": {"worker_services": ["inference/prefill-short"], "admission": {"max_pending_prefill_tokens": 32768}},
|
||||
"decode": {"worker_services": ["inference/decode-short"], "admission": {"max_kv_usage": 0.9}}
|
||||
},
|
||||
{
|
||||
"id": "long-context",
|
||||
"rank": 1,
|
||||
"max_context_tokens": 131072,
|
||||
"prefill": {"worker_services": ["inference/prefill-long"]},
|
||||
"decode": {"worker_services": ["inference/decode-long"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A request with 4k input tokens and a 16k expected peak cannot fit the short
|
||||
bucket's context capacity. It selects the long bucket and both of its P/D groups.
|
||||
With a known peak of 8k or less, the same input selects both groups of the short bucket.
|
||||
|
||||
Membership has two mutually exclusive forms. `worker_ids` pins exact discovery
|
||||
identities: URLs for static discovery, `namespace/pod-UID` for Kubernetes pods.
|
||||
`worker_services` lists Kubernetes `namespace/service` names and matches any
|
||||
listed Service through its EndpointSlices' `kubernetes.io/service-name` label.
|
||||
Service membership survives pod replacement and scale-up; worker IDs still
|
||||
change per pod incarnation so breaker, load and affinity state cannot carry over.
|
||||
Only Services included by the discovery watch are visible. Multiple slices and
|
||||
Services selecting one pod contribute the union of its memberships; removing a
|
||||
slice removes only memberships no longer advertised by another slice. Membership
|
||||
updates preserve the live worker and its in-flight state. Readiness and routing
|
||||
use the same group filter, including model, health and role constraints.
|
||||
|
||||
Omitted group fields mean every engine of the group's role and `--policy`
|
||||
(power-of-two on decode). Admission limits a group leaves unset take
|
||||
`--max-in-flight` / `--max-kv-usage`, field by field. JSON `null` also inherits;
|
||||
there is no per-group disable value. To limit only some groups, leave the CLI
|
||||
default unset and configure those groups explicitly. A group policy is
|
||||
`power_of_two` or the `--policy` kind, whose affinity, cache and tokenizer
|
||||
settings are the ones resolved. Startup rejects empty or duplicate IDs, a bucket
|
||||
that is not exactly plain or P/D, inverted token ranges, a minimum input above
|
||||
the context capacity, empty membership lists, blank worker IDs, malformed
|
||||
Service names, both membership fields together, non-positive capacity or SLO
|
||||
estimates, usages outside (0, 1], zero count limits, and a policy a stage cannot
|
||||
serve. Without the flag, reorg builds one plain and one P/D bucket
|
||||
over all engines. The worker registry still rejects mixed plain and PD engines
|
||||
within one model.
|
||||
|
||||
Legacy `--bucket-config` files describe role-specific buckets and do not
|
||||
translate: P/D associations must be written explicitly, never inferred from
|
||||
rank or names.
|
||||
|
||||
Retained settings keep their meanings, defaults, units, and validation unless a
|
||||
change is listed below. Policy-specific tuning applies to role groups using that
|
||||
policy. Reject unsupported settings and incompatible combinations at startup;
|
||||
do not accept and ignore them.
|
||||
|
||||
### Retained behavior
|
||||
|
||||
- Keep `load_based` as the CLI name for `LeastLoadPolicy`.
|
||||
- Preserve explicit engine membership and context constraints. Bucket-level
|
||||
ranges and ordering replace independent per-stage selection.
|
||||
SLO preferences order whole buckets; deployment configuration remains explicit
|
||||
until the production configuration factory and serving switchover are ready.
|
||||
- Preserve cache-provider selection, endpoint validation, query timeout and
|
||||
concurrency limits, and unavailable-backend fallback.
|
||||
- Preserve the 1,024-token default minimum cache hit, optional ratio gate,
|
||||
and candidate bounds. Shared affinity modes replace cache diversion knobs.
|
||||
- Preserve session and sticky headers, idle timeouts, eviction cadence, and the
|
||||
four sticky fallback choices. Global modes need a bucket-first migration design.
|
||||
- Map `--filter overloaded` and `--max-in-flight` to `max_inflight_requests`;
|
||||
the existing router-local counter remains the source. `--max-kv-usage` sets
|
||||
`max_kv_usage` for groups that leave it unset.
|
||||
- Preserve configured capacity, pending-prefill, and in-flight checks, including
|
||||
their missing-report behavior. Power-of-two applies admission to its selected
|
||||
engine; other policies explicitly place checks in their selection logic.
|
||||
Other paths use `AdmissionLimits::default()` unless a check is configured.
|
||||
Never silently discard a configured budget.
|
||||
|
||||
Listener and shutdown configuration, discovery, worker health and circuit
|
||||
breakers, tokenizer loading, request timeouts, sampling overrides, and logging
|
||||
remain outside the policy redesign. Preserve their existing behavior and
|
||||
validation, including dispatch-time breaker probes and request cancellation.
|
||||
|
||||
### Deliberate changes
|
||||
|
||||
| Behavior | Target |
|
||||
| --- | --- |
|
||||
| Power-of-two admission | Check only the chosen engine; rejection advances to the next bucket |
|
||||
| Round-robin cursor | One cursor per role-group policy instance |
|
||||
| Capacity exhaustion | Try the next compatible bucket; return accumulated rejection details if all fail |
|
||||
| Primary/backup proposals and post-policy substitution | Removed; each policy returns one engine |
|
||||
| Session affinity | Reuse or replace bindings through shared affinity modes |
|
||||
| Omitted `--affinity-mode` | Defaults to `prefer` for both affinity policies |
|
||||
| Pressure-guard tuning | Replaced by `--affinity-load-factor` and `--affinity-load-gap` |
|
||||
| Policy attachment | Explicit role-group policy overrides the applicable model/stage default |
|
||||
|
||||
Reject these dropped options explicitly:
|
||||
|
||||
- `--policy fused_score` and `--policy score_policy`, including `--fuse` terms
|
||||
and weights.
|
||||
- `--decode-policy legacy_host_affinity`.
|
||||
- `--stable-pair`.
|
||||
- `--affinity-mode soft` / `strict`; use `prefer` / `balanced`.
|
||||
- Legacy pressure guards, cache switch margins, queue limits and saturation floors.
|
||||
- `--filter prefix_cache` and `--prefix-cache-min-share`. The removed prefix-share
|
||||
filter is not equivalent to the cache-aware minimum-hit gate.
|
||||
|
||||
## 8. Dispatch and failure handling
|
||||
|
||||
A successful pick means admission passed against the observed state. Health and
|
||||
capacity can change before dispatch. The handler owns network operations, retry
|
||||
rules, and accounting for the engines actually dispatched to.
|
||||
|
||||
Selection fallback advances through the ordered compatible buckets before any
|
||||
network dispatch. Every PD attempt picks both engines from that bucket. With
|
||||
`--retry-max-attempts` above 1, a dispatch that fails before any response
|
||||
reaches the client reruns selection from the first bucket with the failed
|
||||
engine excluded (`BucketRequest::excluded`); it never replaces a pick whose
|
||||
response the client has started to receive.
|
||||
|
||||
For PD, acquire and release accounting for the actual stages and clean up
|
||||
partial setup on failure. Policy selection does not own the PD request lifetime.
|
||||
Selection metrics must not imply that dispatch or execution succeeded.
|
||||
|
||||
## Implementation status
|
||||
|
||||
This PR adds the side-by-side interfaces in `src/buckets_reorg.rs` and
|
||||
`src/policies_reorg/`, plus a configurable bucket-first implementation behind
|
||||
`chat_completions`. The live `src/policies/` path remains the default.
|
||||
|
||||
Implemented here:
|
||||
|
||||
- `BucketResolver::resolve` returns all length-compatible buckets in optional
|
||||
SLO-preference tiers, with capacity/rank/ID order within each tier.
|
||||
- `Bucket::pick_engines` owns plain/PD orchestration and stage-specific policy
|
||||
requests; `BucketRequest` carries prepared facts and `BucketPick` retains picks.
|
||||
- `Bucket` owns input limits, context capacity, rank, and plain-or-PD groups.
|
||||
- `EngineGroup::pick` owns live candidate filtering, policy invocation, and
|
||||
exact candidate validation, without cross-bucket fallback.
|
||||
- `Policy::pick`, within-group fallback interface, per-engine `EngineAdmission::check`,
|
||||
and `AdmissionLimits` with running and KV usage shares plus waiting,
|
||||
pending-prefill and in-flight counts. Power-of-two samples two distinct engines, compares stage pressure,
|
||||
and checks its selected engine with no replacement on rejection.
|
||||
- Policy-owned load dependency and local observations. Power-of-two passes the
|
||||
selected engine's load record directly to admission, without another snapshot.
|
||||
`PickRequest`, `Pick`, and bucket APIs carry no load observations.
|
||||
- The reorg chat implementation iterates resolved buckets, calls `pick_engines`,
|
||||
advances on empty candidates/admission rejection, and
|
||||
dispatches only after one complete selection. Exhaustion retains admission reasons.
|
||||
- `AppContext::chat_routing` configures legacy versus reorg routing on the same
|
||||
endpoint and carries the reorg model-resolver map.
|
||||
- Optional bucket TTFT/throughput estimates and preferences, with enabled-header
|
||||
parsing and whole-bucket fallback in the shared chat route.
|
||||
- `CacheAwarePolicy` reads local radix-tree or remote indexer prefixes, intersects
|
||||
exact worker URLs with the current group, applies hit thresholds and candidate
|
||||
bounds, and applies the shared affinity mode.
|
||||
- `PrefixMemo` shares lookup results (including misses and unavailable backends)
|
||||
across bucket attempts for one prepared request. Entries are keyed by the shared
|
||||
`Arc<CacheSource>` so different index namespaces remain independent. Each pick
|
||||
reruns its own candidate filtering and admission after obtaining a fresh snapshot.
|
||||
- Cache selection checks bounded candidates explicitly; rejected owners are
|
||||
excluded from fallback. The cache policy checks its fallback winner's admission.
|
||||
Cache policies require plain/prefill groups.
|
||||
- `SessionAwarePolicy` reuses admitted model/bucket/role-scoped bindings from a
|
||||
shared `AffinityStore`, falling back to power-of-two for new or keyless sessions.
|
||||
Assignments follow admission; concurrent binding winners are rechecked.
|
||||
Rejected affinity invokes fallback; failed fallback preserves the binding.
|
||||
The caller owns expiry and sweeper lifecycle. A binding may remain after a
|
||||
later PD group fails, because it records placement rather than dispatch.
|
||||
|
||||
- `--bucket-config` builds explicit plain and P/D buckets with per-group
|
||||
membership, policy and admission; omitting it builds the default buckets.
|
||||
|
||||
Follow-up work includes remaining selection policies.
|
||||
|
||||
Not yet implemented in the reorg path:
|
||||
|
||||
- Other concrete selection policies.
|
||||
- Global session modes and sticky routing-key affinity.
|
||||
- Power-of-k cache-miss fallback configuration and cache decision metrics.
|
||||
- Shared load interpretation, dispatch correction, and policy-specific
|
||||
dispatch-timestamp requirements.
|
||||
- PD compatibility filtering and legacy-route switchover.
|
||||
|
||||
The preceding policy sections describe target behavior for those follow-ups;
|
||||
they do not claim those capabilities are present in this PR.
|
||||
+289
-674
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,640 @@
|
||||
# sgl-router reference
|
||||
|
||||
Detailed behavior of the features summarized in [README.md](README.md). Routing
|
||||
design, policies and bucket configuration are in [ARCHITECTURE.md](ARCHITECTURE.md).
|
||||
|
||||
- [Discovery and KV signals](#discovery-and-kv-signals)
|
||||
- [Bucket file membership](#bucket-file-membership)
|
||||
- [Routing and reliability](#routing-and-reliability)
|
||||
- [Chat rendering](#chat-rendering)
|
||||
- [Native `/generate`](#native-generate)
|
||||
- [OpenAI over /generate](#openai-over-generate)
|
||||
- [Embeddings](#embeddings), [Classify](#classify), [Completions](#completions), [Rerank](#rerank)
|
||||
- [Anthropic Messages](#anthropic-messages)
|
||||
- Per-model rendering: [DeepSeek V4 and V4.1](#deepseek-v4-and-v41), [GLM-5.3](#glm-53), [Qwen3.8](#qwen38), [Kimi-K3](#kimi-k3)
|
||||
- [HTTP/2](#http2)
|
||||
- [Native gRPC](#native-grpc)
|
||||
- [Upgrading from `cache_aware_zmq`](#upgrading-from-cache_aware_zmq)
|
||||
|
||||
## Discovery and KV signals
|
||||
|
||||
### PD version groups
|
||||
|
||||
To run two engine versions side by side in PD mode (e.g. during a rollout),
|
||||
pass `--pd-version-group-label <key>`. A prefill worker is then paired only with
|
||||
decode workers that have the same value for that EndpointSlice label (inherited
|
||||
from the Service, so use one Service per role and version), so KV never crosses
|
||||
versions. Unlabeled workers form their own group.
|
||||
|
||||
```bash
|
||||
sgl-router --model-id qwen3 --service-discovery \
|
||||
--prefill-selector app=engines-qwen3,role=prefill \
|
||||
--decode-selector app=engines-qwen3,role=decode \
|
||||
--pd-version-group-label sglang.ai/version-group
|
||||
```
|
||||
|
||||
### External KV indexer
|
||||
|
||||
```bash
|
||||
sgl-router \
|
||||
--model-id qwen3 \
|
||||
--tokenizer-path /models/qwen3/tokenizer.json \
|
||||
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
|
||||
--policy cache_aware \
|
||||
--cache-prefix-provider indexer \
|
||||
--kv-indexer-endpoint http://10.0.0.10:50051 \
|
||||
--kv-indexer-query-timeout-ms 100 \
|
||||
--kv-indexer-query-max-inflight 32
|
||||
```
|
||||
|
||||
The Indexer replaces the Router-local radix tree as the native Cache-Aware
|
||||
signal. Query timeouts and local concurrency are bounded by the two Indexer
|
||||
options, which default to 100 ms and 32 respectively.
|
||||
|
||||
### LoRA cache identity
|
||||
|
||||
LoRA routing requires workers that publish `BlockStored.lora_name`. Requests
|
||||
select the adapter through `model: "base:adapter"` on OpenAI routes, or
|
||||
`lora_path` on native routes. Salted generation requests also carry `cache_salt`.
|
||||
|
||||
Worker block IDs are opaque and may differ across adapter loads or workers.
|
||||
The router derives adapter-aware lookup keys from event tokens and keeps a
|
||||
per-worker, per-DP-rank mapping for parent links and removals. Multiple block
|
||||
IDs can own the same lookup key; removing one leaves the other owners and
|
||||
storage tiers intact. The remote indexer bridge performs the same translation.
|
||||
Base-model events retain their existing published-key path.
|
||||
|
||||
New replicas request `?include_lora=true` snapshots containing LoRA mappings
|
||||
(version 2). Legacy consumers receive a base-only version-1 view, so mixed
|
||||
snapshots do not prevent base-cache bootstrap during rolling upgrades. Base-only
|
||||
snapshots retain version 1 for every consumer. LoRA stores buffered during bootstrap retain their
|
||||
tokens; the original base-model buffering and replay behavior is unchanged.
|
||||
|
||||
Reloading different adapter weights under the same public name can still
|
||||
credit stale engine blocks until their eviction. Names identify adapters;
|
||||
versioned names are needed to distinguish different weights.
|
||||
|
||||
### KV-event gap replay
|
||||
|
||||
ZMQ drops KV events at the publisher's high-water mark, and a lost removal
|
||||
leaves the router crediting a worker with blocks it no longer holds. Give each
|
||||
engine a replay socket and the router re-fetches the batches a sequence gap
|
||||
skipped:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server ... \
|
||||
--kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558"}'
|
||||
```
|
||||
|
||||
Rank r publishes on 5557 + r and replays on 5558 + r, so with `dp_size > 1`
|
||||
place the replay base at least dp_size ports past the PUB base. The engine keeps
|
||||
the last `buffer_steps` batches (default 10000); older gaps stay unrepaired.
|
||||
`sgl_router_kv_event_replays_total{outcome}` counts the attempts.
|
||||
|
||||
### Pending prefixes
|
||||
|
||||
Until the engine's KV events arrive, requests that share a cold prefix see no
|
||||
cached owner and spread across workers, each prefilling the same prefix. This
|
||||
is common with parallel sampling, RL rollouts and agent fan-out.
|
||||
`--cache-pending-prefix-ttl-ms` credits a worker with a prompt's prefix for that
|
||||
long after routing it there, so the burst stays together. It is off by default
|
||||
and needs `--policy cache_aware` with the Router-local radix tree.
|
||||
`sgl_router_cache_pending_prefix_hits_total` counts lookups where a pending
|
||||
prefix matched deeper than any confirmed one; the policy's candidate and
|
||||
admission filters still decide whether that worker is picked.
|
||||
|
||||
### Peer bootstrap (Kubernetes)
|
||||
|
||||
A replica that starts mid-fleet subscribes to each worker's KV topic
|
||||
mid-stream, so everything already resident in the engines' caches is invisible
|
||||
to it and it routes cache-blind until traffic re-stores those blocks —
|
||||
degrading the engines' locality for the whole fleet, once per replica on every
|
||||
rolling update. With a peer selector set, a booting replica instead pulls a
|
||||
tree snapshot from a warm sibling over `/internal/kv_snapshot` and splices it
|
||||
under its own live delta stream, and `/readyz` stays 503 until that bootstrap
|
||||
settles. Off unless `--kv-peer-selector` is set; requires `--policy
|
||||
cache_aware` with the Router-local radix tree (not an external Indexer) and
|
||||
`--service-discovery`, which the CLI enforces.
|
||||
|
||||
```bash
|
||||
sgl-router \
|
||||
--model-id qwen3 --policy cache_aware \
|
||||
--service-discovery --service-discovery-namespace prod \
|
||||
--selector app=engines-qwen3 \
|
||||
--kv-peer-selector kubernetes.io/service-name=sgl-router
|
||||
```
|
||||
|
||||
- `--kv-peer-selector` is matched against **EndpointSlice labels** — the
|
||||
router Service's labels plus `kubernetes.io/service-name`, never the pods'
|
||||
labels, so `kubernetes.io/service-name=<router-service>` is the unambiguous
|
||||
choice. A pod-template label matches nothing and every replica boots cold.
|
||||
- `--kv-bootstrap-timeout-ms` (default 600000) bounds the whole bootstrap;
|
||||
readiness waits on it, so startup/readiness probes must tolerate a replica
|
||||
staying unready this long.
|
||||
- `--kv-bootstrap-fetch-timeout-cap-ms` (default 300000) caps one snapshot
|
||||
fetch within that budget.
|
||||
- `--kv-bootstrap-seed-required` (off by default) keeps holding `/readyz` when
|
||||
siblings were present but their tree could not be pulled, bounded at
|
||||
max(3× the bootstrap timeout, 60s) — it delays a failed seed's replica (and
|
||||
with it a rolling update) rather than shipping it cache-blind; it cannot
|
||||
stall a rollout indefinitely.
|
||||
|
||||
The deployment needs two things beyond the flags: the router's ServiceAccount
|
||||
must hold `get`/`list`/`watch` on `endpointslices` in the watched namespace
|
||||
(the worker-discovery Role already grants this), and the pod spec should wire
|
||||
`POD_NAME`, `POD_NAMESPACE` and `POD_IP` from the downward API so a replica
|
||||
can exclude itself from its own peer list. A rolling update should surge
|
||||
(`maxUnavailable: 0`) so new replicas always have warm siblings to copy from.
|
||||
`tests/e2e/k8s_integration/manifests/kv-bootstrap.yaml` is a worked example of
|
||||
all of it, and the `sgl_router_kv_bootstrap_*` series in
|
||||
[monitoring/README.md](monitoring/README.md) make the whole path observable.
|
||||
|
||||
## Bucket file membership
|
||||
|
||||
`worker_services` matches Kubernetes Services by `namespace/name`, using the
|
||||
`kubernetes.io/service-name` label on watched EndpointSlices. Replacement pods
|
||||
and new replicas join the same group automatically. The router's discovery
|
||||
selectors must include those EndpointSlices; this field does not expand the watch.
|
||||
A worker selected by several Services belongs to each of them.
|
||||
|
||||
For static URL discovery, use `"worker_ids": ["http://worker:30000"]`: each ID
|
||||
is the configured worker URL. Kubernetes worker IDs are `namespace/pod-UID`
|
||||
(and change when a pod is replaced), so use `worker_services` for durable pools.
|
||||
Set only one membership field, or omit both for every engine of the group's role.
|
||||
Empty membership lists and blank worker IDs are rejected at startup.
|
||||
|
||||
Admission fields left unset or set to `null` inherit CLI defaults. To apply a
|
||||
limit only to selected groups, omit that CLI default and set it on those groups.
|
||||
|
||||
## Routing and reliability
|
||||
|
||||
### Optional tokenizer for load-only routing
|
||||
|
||||
`--no-tokenizer` skips tokenizer loading for load-only policies such as
|
||||
`power_of_two` and `session_aware`, on either routing path. Workers tokenize the
|
||||
original messages, and `/v1/tokenize` and `/v1/detokenize` are unavailable.
|
||||
Cache-aware routing, prefix-cache terms or filters, and buckets with token-length
|
||||
or context limits require a tokenizer. Reorg buckets that only select membership
|
||||
and load-based policies can use `--no-tokenizer`.
|
||||
|
||||
### DP-rank routing
|
||||
|
||||
An engine launched with `--dp-size` or `--attn-dp-size` runs several DP ranks,
|
||||
each with its own KV cache, behind one endpoint. With `--dp-aware`, the router
|
||||
also picks the rank inside the selected worker. It sends that rank as
|
||||
`X-Data-Parallel-Rank`, which the engine honors, and overwrites any value the
|
||||
client sent. The router picks the first of these that applies:
|
||||
|
||||
1. A hash of the sticky routing key or session id, so a conversation keeps
|
||||
its rank and router replicas agree.
|
||||
2. The rank with the deepest cached prefix in the local KV tree.
|
||||
3. The rank with the fewest requests this router has in flight on it, or with
|
||||
`--dp-rank-policy round_robin`, the worker's ranks in turn.
|
||||
|
||||
In PD mode, decode is ranked by step 3 only. The decode request names the
|
||||
prefill rank in `disagg_prefill_dp_rank`, and the bootstrap room satisfies
|
||||
`room % prefill_dp_size == prefill_rank`.
|
||||
|
||||
### Retries
|
||||
|
||||
`--retry-max-attempts N` (default 1, which disables retries; 3 is typical) lets
|
||||
a request that fails before any response reaches the client be sent again to a
|
||||
worker it has not tried yet. A failure is a transport error, an open circuit
|
||||
breaker, a 5xx, a 429, or a timeout: `--request-timeout-secs` bounds a whole
|
||||
non-streaming response, and a streaming one until its headers arrive. SGLang's
|
||||
chat endpoint sends those headers with the first token, so for streaming chat
|
||||
the same flag bounds time to first token, queueing included; raise it for
|
||||
long-context or deeply queued fleets. Once a streaming response's 2xx status
|
||||
has been sent, it is never retried, and neither is any other 2xx or 4xx. In PD mode the failed
|
||||
side is excluded, or both sides when the caller set `rid`, which an engine still
|
||||
running it would refuse; a new pair gets a new bootstrap room. Retries share the
|
||||
request's `--stale-request-timeout-secs` deadline, and none starts after it. When
|
||||
every eligible worker has failed, the client gets the last failure.
|
||||
`sgl_router_retries_total` counts the retried attempts.
|
||||
|
||||
Each retry first waits out a jittered exponential backoff: a random delay in
|
||||
`[d/2, d]`, where `d` starts at `--retry-initial-backoff-ms` (default 50),
|
||||
doubles per retry, and is capped at `--retry-max-backoff-ms` (default 2000).
|
||||
Set the initial backoff to 0 to retry immediately. A backoff never waits past
|
||||
the request's stale deadline.
|
||||
|
||||
### Engines with `--api-key`
|
||||
|
||||
The router reads each worker's `/server_info` and `/model_info` to learn its
|
||||
model, KV-event publisher, HTTP/2 support and DP size. An engine launched with
|
||||
`--api-key` rejects those requests without the key, so pass the same key as
|
||||
`--worker-api-key`. Chat requests and `/flush_cache` forward the caller's
|
||||
`Authorization` header instead, so callers still need the engine key.
|
||||
|
||||
### Fleet-wide sampling contract
|
||||
|
||||
`--override-sampling-params` fixes the sampling configuration for every client
|
||||
of this router, independently of what the engine's own defaults happen to be
|
||||
(on native [`/generate`](#native-generate), in each prompt's `sampling_params`):
|
||||
|
||||
```bash
|
||||
sgl-router \
|
||||
--model-id qwen3 \
|
||||
--tokenizer-path /models/qwen3/tokenizer.json \
|
||||
--worker-urls http://10.0.0.1:30000 \
|
||||
--override-sampling-params '{"temperature": 1, "top_p": 0.95, "n": 1}' \
|
||||
--sampling-param-conflict reject
|
||||
```
|
||||
|
||||
It takes one JSON object keyed by the request-body field names
|
||||
(`temperature`, `top_p`, `top_k`, `min_p`, `repetition_penalty`,
|
||||
`frequency_penalty`, `presence_penalty`, `n`). A configured value is injected
|
||||
whenever the request omits that field, so the engine's defaults cannot drift
|
||||
from what the operator declared. `temperature`, `top_p`, `top_k`, `min_p` and
|
||||
`repetition_penalty` are the five the engine resolves from the model's own
|
||||
`generation_config`, which is what makes them drift when a deployed image
|
||||
changes; the rest have fixed API defaults.
|
||||
|
||||
An explicit `null` counts as omitting the field, not as a client-supplied
|
||||
value: the OpenAI API types these parameters as nullable with a documented
|
||||
default, so `null` asks for the default — and on a governed fleet the
|
||||
configured value is what the default is.
|
||||
|
||||
For a request that does send a value, `--sampling-param-conflict` decides:
|
||||
`reject` (the default) 400s a differing value before admission, while `allow`
|
||||
forwards the client's value untouched — the router never silently rewrites what
|
||||
a client sent. A `reject` response carries
|
||||
`x-router-error-code: sampling_contract_violation` and is counted in
|
||||
`sgl_router_sampling_contract_rejections_total{param}`, so a rollout's blast
|
||||
radius is visible per parameter rather than folded into `bad_request`.
|
||||
|
||||
`reject` also 400s a value it cannot read as a number, on a governed parameter
|
||||
only. The engine coerces more than JSON numbers — a bool and a numeric string
|
||||
both become numbers — by rules that are undocumented and need not match across
|
||||
a fleet, so a value the router cannot read is one it cannot prove conforms, and
|
||||
waving it through would make the pin bypassable. The common coercions are
|
||||
matched exactly (`false` is 0, `"0.5"` and `"1_0"` are 0.5 and 10), so this
|
||||
refuses only genuine garbage. `allow` is unaffected: it promises nothing, so
|
||||
such a value keeps flowing to the engine, which owns the request schema.
|
||||
|
||||
A value may also be an inclusive band, `{"min": LO, "max": HI}`, for a
|
||||
parameter that stays tunable inside a range. A band names no value to inject,
|
||||
so it constrains only the requests that name the parameter; one that omits it
|
||||
gets the model's own `generation_config` default, which the router cannot see.
|
||||
Because a band can only ever reject, combining one with `allow` is a startup
|
||||
error.
|
||||
|
||||
Values are range-checked at startup, so a misconfiguration fails the launch
|
||||
instead of 400ing every request at the engine. Repeating a key in the flag is
|
||||
also a startup error, rather than silently enforcing whichever copy came last.
|
||||
|
||||
| parameter | accepted | notes |
|
||||
| --- | --- | --- |
|
||||
| `temperature` | `[0, 2]` | |
|
||||
| `top_p` | `(0, 1]` | |
|
||||
| `top_k` | `>= 1`, or exactly `-1` | `-1` is the engine's "whole vocabulary" spelling and its default. Being non-contiguous it cannot bound a band. Note `top_k: 1` is greedy decoding, not "disabled". |
|
||||
| `min_p` | `[0, 1]` | not an OpenAI parameter; the engine's domain |
|
||||
| `repetition_penalty` | `(0, 2]` | not an OpenAI parameter; the engine's domain |
|
||||
| `frequency_penalty` | `[-2, 2]` | |
|
||||
| `presence_penalty` | `[-2, 2]` | |
|
||||
| `n` | `[1, 128]` | |
|
||||
|
||||
The OpenAI domains are deliberately narrower than what the engine itself
|
||||
accepts (`SamplingParams.verify` would take `temperature: 5`): these values are
|
||||
injected into request bodies, and a fleet contract outside the range every
|
||||
OpenAI client library validates against is far more likely a typo than an
|
||||
intent.
|
||||
|
||||
Cost: a request that named every configured field forwards its original bytes
|
||||
untouched. One that omits a field has the scalars spliced directly into the
|
||||
body bytes — no parse, no re-serialize — which on a 16 MiB body is ~0.24 ms
|
||||
against ~5.7 ms for a `serde_json` round-trip. Only `input_ids` and PD
|
||||
bootstrap injection still parse and re-serialize, because those may have to
|
||||
overwrite a key the client sent.
|
||||
|
||||
### Relationship to the engine's own `--preferred-sampling-params`
|
||||
|
||||
The engine has an inject-when-absent flag of its own,
|
||||
`--preferred-sampling-params`, merged in
|
||||
`python/sglang/srt/managers/tokenizer_manager.py` as
|
||||
`{**preferred, **obj.sampling_params}`. On `/v1/chat/completions` it is
|
||||
currently a no-op: `ChatCompletionRequest.to_sampling_params`
|
||||
(`python/sglang/srt/entrypoints/openai/protocol.py`) resolves every sampling
|
||||
key eagerly through `generation_config` and then its own defaults, so the
|
||||
right-hand side of that merge is always fully populated and always wins.
|
||||
(`/v1/responses`, in the same file, already omits `None` entries for exactly
|
||||
this reason.) If that is fixed engine-side, `--preferred-sampling-params`
|
||||
covers the inject-when-absent half for a single engine.
|
||||
|
||||
What stays the router's to own either way is the enforcement half: the
|
||||
`reject` immutability contract with its 400 before admission — costing no
|
||||
queue slot and no engine round-trip — the `sampling_contract_violation` code
|
||||
and per-parameter counter, bands, and one contract applied at a shared ingress
|
||||
across engines whose own flags the router operator may not control.
|
||||
|
||||
## Chat rendering
|
||||
|
||||
The router renders chat requests with dynamo-render (`dynamo-renderer`): the model's
|
||||
HF Jinja template from `tokenizer_config.json` or a sibling
|
||||
`chat_template.jinja`, or dynamo-render's built-in DeepSeek encoder (V4 family, V3.2)
|
||||
for template-less models. Cache-aware routing hashes the rendered tokens so its
|
||||
prefix queries match the blocks the engine caches. Models the engine encodes in
|
||||
code but dynamo-render cannot tokenize here (Inkling) route via raw prompt
|
||||
text, as does any model whose template fails to load or render.
|
||||
|
||||
Some chats additionally forward the rendered tokens to the engine as
|
||||
`input_ids`, retaining the original messages, so the engine skips
|
||||
re-tokenizing. How many depends on the model's renderer. DeepSeek-V4's, V4.1's
|
||||
and Kimi-K3's native encoders and GLM-5.3's and Qwen3.8's templates are
|
||||
fixture-verified against SGLang's request normalization, so they forward every
|
||||
chat except multimodal ones and those with caller-provided `input_ids`. Renderers
|
||||
without that verification (HF Jinja templates)
|
||||
forward only plain text chat requests (string `content`, no tools, no template
|
||||
kwargs or reasoning controls or historical `reasoning_content`, no assistant
|
||||
continuation, no consecutive users or non-leading system turns) and warn
|
||||
`UNVERIFIED` at startup; every other request shape is rendered for routing
|
||||
only.
|
||||
|
||||
Use matching model files on the router and workers, and set
|
||||
the same `--default-chat-template-kwargs`, `SGLANG_DEFAULT_THINKING`,
|
||||
`SGLANG_DSV4_REASONING_EFFORT`, and `SGLANG_DSV41_REASONING_EFFORT` on both.
|
||||
The router reads these render defaults from its own configuration and environment;
|
||||
it does not discover the workers' settings. Point `--tokenizer-path` at the workers'
|
||||
model snapshot so the V4 effort profile is read from the same
|
||||
`encoding/encoding_dsv4.py`.
|
||||
|
||||
Set `--disable-input-ids-forwarding` for this router's model when worker-side
|
||||
rendering has not been verified to match. This disables router-generated IDs
|
||||
for every routing policy; cache-aware routing still renders and tokenizes
|
||||
locally, and the original messages reach the workers for engine processing.
|
||||
Caller-supplied `input_ids` remain caller-owned and pass through unchanged.
|
||||
|
||||
Forwarding logs its assumptions at startup. In particular, disable it when the
|
||||
router's render defaults differ from the workers', for worker template overrides
|
||||
not reflected in the router's model files, for worker parser overrides such as
|
||||
`--tool-call-parser deepseekv32` that select a native encoder over a shipped
|
||||
template, or conversation templates with stop strings (the engine's `input_ids`
|
||||
path skips those template stops). Disabling forwarding preserves engine behavior
|
||||
but does not establish parity for local routing hashes.
|
||||
|
||||
Also set `--disable-input-ids-forwarding` for array-only templates: Dynamo may wrap
|
||||
string content into arrays differently from the worker. The pinned Dynamo renderer does not expose
|
||||
its conversion flag, so the router cannot automatically block these templates.
|
||||
Detailed content-format parity coverage follows in #39133.
|
||||
|
||||
The Dynamo crates are pinned exactly and `Cargo.lock` is committed; CI builds
|
||||
with `--locked`, so rendered bytes cannot change without a reviewed diff.
|
||||
|
||||
Router tokenization sits on the TTFT path for every chat it renders. Two opt-in
|
||||
flags make it cheaper:
|
||||
|
||||
- `--tokenizer-backend fast` encodes with fastokens (decoding stays on HF). It
|
||||
needs a `tokenizer.json` and falls back to `hf` when fastokens cannot load it.
|
||||
- `--tokenizer-l1-cache-mb N` caches prefix tokenizations at special-token
|
||||
boundaries, so a multi-turn chat encodes only the turns added since the
|
||||
previous request. Boundaries are unconditional, non-normalized, non-stripping
|
||||
special tokens with no overlapping added-token spellings. Unsafe candidates
|
||||
are excluded; if none remain, encoding proceeds without the cache.
|
||||
|
||||
On a ~69K-token DeepSeek-V4 chat, `hf` encodes in ~40 ms, `fast` in ~4 ms, and
|
||||
a new turn on a cached history in ~0.2 ms. The DeepSeek fixtures check every
|
||||
case under `hf`, `fast`, and `fast` with L1. Startup logs report the resolved
|
||||
backend and cache state. `/metrics` exposes only
|
||||
`sgl_router_tokenizer_l1_tokens_total{source="cached"|"encoded"}` to measure
|
||||
how much tokenization work the cache reuses.
|
||||
|
||||
## Native `/generate`
|
||||
|
||||
`/generate` (`POST` or `PUT`) has the engine's interface: the same
|
||||
`GenerateReqInput` body and the same response, buffered or SSE. The body names no
|
||||
model, so requests go to the one this router serves. Worker selection
|
||||
(`--chat-routing`, `--policy`), PD dispatch, and abort-on-disconnect are shared
|
||||
with chat completions.
|
||||
|
||||
With a tokenizer loaded, the router tokenizes `text` (a string or a list) with
|
||||
the special tokens SGLang adds (BOS per `add_bos_token` for Llama-, Gemma- and
|
||||
Cohere-class tokenizers, otherwise the `tokenizer.json` post-processor's), and
|
||||
forwards it as `input_ids`. The engine skips tokenizing and routing sees its exact
|
||||
tokens. Multimodal requests keep `text`, since the engine expands placeholders
|
||||
from it, and `--disable-input-ids-forwarding` keeps it for every request. So does
|
||||
a model whose `tokenizer.json` normalizer transformers replaces on load (legacy
|
||||
SentencePiece Llama files, bge-m3), or whose `tokenizer_config.json` the router
|
||||
cannot read, as the router cannot reproduce its tokens. A batch goes to one
|
||||
worker: load counts every prompt and each of its `n` samples, while bucket and
|
||||
context limits bound the longest prompt plus its own `max_new_tokens`.
|
||||
|
||||
Otherwise the body passes through, plus PD bootstrap fields, a minted `rid` for
|
||||
a single prompt that has none, and `--override-sampling-params` defaults. Under
|
||||
`--dp-aware` each worker's body also carries the chosen `routed_dp_rank`; a PD
|
||||
batch or `n > 1` request leaves the prefill rank to the engine, which gives item
|
||||
`i` the bootstrap room `room + i`.
|
||||
|
||||
## OpenAI over /generate
|
||||
|
||||
`/v1/completions` and `/v1/chat/completions` behave as the engine's own routes,
|
||||
but the router runs SGLang's OpenAI layer itself (`sglang-processor`'s `openai` module): it lowers
|
||||
the request to the `GenerateReqInput` SGLang would build, sends it to
|
||||
`/generate` like a native request, and builds the OpenAI response, buffered or
|
||||
SSE, from the engine's output. The engine's `/server_info` supplies the server
|
||||
args that layer reads (`--enable-cache-report`,
|
||||
`--stream-response-default-include-usage`, `--incremental-streaming-output`, the
|
||||
custom-labels header). Requests it cannot reproduce exactly go to the engine's
|
||||
own route as sent: workers that disagree on those args or never
|
||||
reported them, `--completion-template`, `return_hidden_states` and other
|
||||
`sglext` outputs, `echo` or `logprobs` without a router tokenizer, and bodies
|
||||
SGLang would reject. Chat also needs a renderer verified against SGLang (the
|
||||
`AllText` scope: DeepSeek-V4, V4.1, GLM-5.3, Kimi-K3 and Qwen3.8 today), router `--default-chat-template-kwargs`
|
||||
equal to the workers', a reasoning parser the processor reproduces
|
||||
(`deepseek-v4`, `deepseek-v41`, `glm45`, `kimi_k3`, `qwen3`) or none, and no media yet. Tools are served with `tool_choice`
|
||||
`auto` or `none`, non-strict, with standard JSON-schema types, and a tool
|
||||
parser the processor reproduces (`deepseekv4`, `deepseekv41`, `kimi_k3`, `qwen3_coder`); `required`, named and strict
|
||||
tools take a constraint from the engine's xgrammar, so they go to the engine
|
||||
(`kimi_k3`'s `auto` tag is reproduced).
|
||||
With `--tool-call-parser glm47`, a chat with tools also goes to the engine: its
|
||||
grammar needs `ebnf_full_assistant`, which `/generate` does not accept.
|
||||
The router reads the model's `config.json` and `generation_config.json` beside
|
||||
`--tokenizer-path` (or from the HF repo), as the engine reads them, so a local
|
||||
path should point into the model's directory. Engine env vars that change the
|
||||
layer (`SGLANG_TOOL_STRICT_LEVEL`, `SGLANG_FORWARD_UNKNOWN_TOOLS`) are taken as
|
||||
unset. `sgl_router_openai_route_total` counts each outcome.
|
||||
|
||||
## Embeddings
|
||||
|
||||
`/v1/embeddings` has the engine's interface: the same OpenAI `EmbeddingRequest`
|
||||
body and response. As for chat completions, `model` must name the served model.
|
||||
A PD fleet answers 400, since prefill and decode engines serve no embeddings.
|
||||
|
||||
Text `input` (a string or a list) is tokenized and forwarded as token IDs, as for
|
||||
[`/generate`](#native-generate), plus the EOS SGLang appends for EmbeddingGemma.
|
||||
Blank prompts and multimodal items stay as sent, for the engine to reject or
|
||||
render. A list is a batch for one worker, routed on load. A single prompt with no
|
||||
`rid` gets a minted one. Under `--dp-aware` the engine picks the rank, since its
|
||||
embeddings endpoint reads none.
|
||||
|
||||
## Classify
|
||||
|
||||
`/v1/classify` has the engine's interface: the same `ClassifyRequest` body and
|
||||
response. It is served as embeddings are, except that a batch of text stays text,
|
||||
since the engine takes token IDs for only one prompt.
|
||||
|
||||
## Completions
|
||||
|
||||
`/v1/completions` has the engine's interface and is prepared like chat (sampling
|
||||
contract, PD bootstrap, DP rank), with the prompt forwarded as sent: tokens the
|
||||
router computes for routing do not replace it.
|
||||
|
||||
## Rerank
|
||||
|
||||
`/v1/rerank` has the engine's interface: the same `V1RerankReqInput` body and
|
||||
response, sent as `POST` or `PUT` to the model this router serves. The body is
|
||||
forwarded as sent, since the engine renders and tokenizes each query-document
|
||||
pair. Routing is on load, with each pair's size estimated from its text. As for
|
||||
embeddings, a PD fleet answers 400 and `--dp-aware` pins no rank.
|
||||
|
||||
## Anthropic Messages
|
||||
|
||||
`/v1/messages` converts each request to a chat completion at ingress, so it is
|
||||
routed exactly like `/v1/chat/completions`, and converts the reply (or stream)
|
||||
back to the Messages format. Consecutive turns of the same role are merged, as
|
||||
the Messages API does, and a final assistant turn is sent as a prefill
|
||||
(`continue_final_message`). Besides the Messages fields, only
|
||||
`chat_template_kwargs` and `continue_final_message` reach the chat request.
|
||||
Every error uses the Anthropic error envelope; 5xx messages are replaced with a
|
||||
generic one, and a failed stream still closes its open block and ends with
|
||||
`message_stop`.
|
||||
|
||||
`/v1/messages/count_tokens` is answered by the router from the model's chat
|
||||
template, without calling an engine. Image tokens are not counted, and a model
|
||||
without a chat template answers 500 rather than a raw-text estimate.
|
||||
|
||||
`thinking` sets the `thinking` / `enable_thinking` chat template kwargs. The
|
||||
engine's own `/v1/messages` also adapts it per reasoning parser (for example
|
||||
`reasoning_effort` for Hunyuan), which the router cannot see; on such models
|
||||
use `output_config.effort` or `chat_template_kwargs` directly.
|
||||
|
||||
## DeepSeek V4 and V4.1
|
||||
|
||||
Native V4 and V4.1 rendering comes from `rust/sglang-processor`, shared with SGLang's Rust
|
||||
server, and follows SGLang's serving path (`serving_chat.py`), not Dynamo's
|
||||
OpenAI defaults: all declared tools are rendered with SGLang's schema
|
||||
defaults, reasoning effort comes from `reasoning` / `reasoning_effort`, and the
|
||||
official/preview effort profile is detected from the checkpoint's
|
||||
`encoding/encoding_dsv4.py` or overridden by `dsv4_reasoning_effort_profile` in
|
||||
`config.json`, as in SGLang. Reference prompts live in `tests/fixtures/deepseek/`
|
||||
and are regenerated by `tests/scripts/generate_deepseek_parity.py`.
|
||||
|
||||
V4.1 Flash uses Dynamo's V4.1 encoder with SGLang's numeric reasoning budgets
|
||||
and tool payloads. Developer messages and media are left to the worker, since
|
||||
the encoder renders them differently.
|
||||
|
||||
## GLM-5.3
|
||||
|
||||
GLM-5.3 and GLM-5.3-Flash render with the checkpoint's `chat_template.jinja` through
|
||||
`rust/sglang-processor`, which builds the template context SGLang's Jinja path builds,
|
||||
so every text chat forwards `input_ids`. Media, a `chat_template` kwarg, `-0` in tools
|
||||
or tool arguments, and U+001C..U+001F in assistant history render on the worker. With
|
||||
the cookbook's parsers, chats without tools are served through `/generate`; chats with
|
||||
tools keep the engine's route (see above).
|
||||
|
||||
## Qwen3.8
|
||||
|
||||
Qwen3.8 (Max, 27B and Flash-Next) renders with the checkpoint's `chat_template.jinja`
|
||||
through `rust/sglang-processor`, which picks it by its template and builds the
|
||||
template context SGLang's Jinja path builds. Every text chat forwards `input_ids`.
|
||||
Media, a `chat_template` key in `chat_template_kwargs`, and text or tool numbers
|
||||
Python's `trim` or `json.dumps` treat unlike Dynamo's (such as `1e-06`) render on the
|
||||
worker. With the cookbook's `qwen3` and `qwen3_coder` parsers, text chats are served
|
||||
through `/generate`; thinking follows the template's `enable_thinking` (on by default).
|
||||
|
||||
## Kimi-K3
|
||||
|
||||
Kimi-K3 renders through `rust/sglang-processor` (dynamo-render's XTML formatter
|
||||
with SGLang's request semantics) and is tokenized with the checkpoint's chunked
|
||||
tiktoken encoding; shapes the formatter renders differently go to the worker.
|
||||
`--tokenizer-path` accepts a
|
||||
local `tiktoken.model` or an HF repo id, whose `tiktoken.model`, `config.json`
|
||||
and `tokenizer_config.json` are downloaded when it has no `tokenizer.json`.
|
||||
With the cookbook's `kimi_k3` parsers, text chats are served through `/generate`.
|
||||
|
||||
## HTTP/2
|
||||
|
||||
There is nothing to configure. The router negotiates per connection inbound and
|
||||
resolves the protocol per worker outbound; every combination below is reached
|
||||
automatically, and HTTP/1.1 remains a supported peer on both edges.
|
||||
|
||||
**Inbound.** The listener accepts cleartext HTTP/2 (h2c, prior knowledge) and
|
||||
HTTP/1.1 on the same `--port`, chosen per connection. A mesh sidecar or
|
||||
load balancer that prefers h2c multiplexes over one connection instead of
|
||||
opening one per request; an HTTP/1.1 client is unaffected.
|
||||
|
||||
**Outbound.** At registration the router reads each worker's `/server_info` and
|
||||
forwards over cleartext h2c only when that worker reports `--enable-http2` (the
|
||||
engine's Granian server, which serves h2c and HTTP/1.1 together) **and** the
|
||||
worker URL is cleartext. Everything else uses the default client, which speaks
|
||||
HTTP/1.1 in cleartext and negotiates ALPN `h2, http/1.1` over TLS:
|
||||
|
||||
| `/server_info` | worker URL | router forwards over |
|
||||
|---|---|---|
|
||||
| `enable_http2: true` | `http://` | cleartext h2c |
|
||||
| `enable_http2: true` | `https://` | HTTP/2 over TLS, by ALPN |
|
||||
| `enable_http2: false`, or absent | any | HTTP/1.1 (cleartext) or ALPN (TLS) |
|
||||
|
||||
The choice is per worker, not fleet-wide, so a mixed fleet works and one
|
||||
worker's state never changes another's. The flag comes from the `/server_info`
|
||||
launch record, which has reported it all along, so no engine change is needed.
|
||||
A worker whose `/server_info` does not answer has no readable protocol and
|
||||
forwards over HTTP/1.1 for as long as it stays registered — a throughput cost,
|
||||
never a correctness one. Admin fan-out (`/flush_cache`) always uses the default
|
||||
client, because it addresses every worker at once rather than a selected one.
|
||||
|
||||
Two things worth knowing when debugging. h2c is prior-knowledge only — there is
|
||||
no negotiation and no fallback — which is why the router requires the engine's
|
||||
own `enable_http2` report before using it. And a worker's protocol is fixed
|
||||
for as long as that worker stays registered: it is read once, at registration,
|
||||
and never re-read. Under the K8s backend a restarting engine flips its
|
||||
EndpointSlice to `ready=false`, which is a `Removed` → `Added` cycle and so a
|
||||
fresh reading; under `--worker-urls` the fan-out happens once at startup and
|
||||
nothing re-registers. So a worker registered over h2c that later stops serving
|
||||
it (a proxy interposed on its port, say) is not detected until it
|
||||
is re-registered; its circuit breaker will open in the meantime.
|
||||
|
||||
## Native gRPC
|
||||
|
||||
`--grpc-port` serves the engine's own gRPC API, `sglang.runtime.v1.SglangService`
|
||||
(`proto/sglang/runtime/v1/sglang.proto`), so a gRPC client can point at the
|
||||
router or at an engine. A call is routed exactly like its HTTP route and sent to
|
||||
the same RPC on the selected engine's native gRPC server, which engines run
|
||||
beside HTTP with their own `--grpc-port`; the router reads that port from each
|
||||
worker's `/server_info`. HTTP requests are unaffected and still reach engines
|
||||
over HTTP.
|
||||
|
||||
| RPC | Routed like |
|
||||
|---|---|
|
||||
| `ChatComplete` | `/v1/chat/completions`: same rendering, `input_ids` forwarding, PD and DP-rank handling |
|
||||
| `Complete` | `/v1/completions` |
|
||||
| `Generate`, `TextGenerate` | `/generate` with `input_ids` or `text` |
|
||||
| `Embed`, `TextEmbed`, `OpenAIEmbed` | `/v1/embeddings` |
|
||||
| `Classify`, `OpenAIClassify` | `/v1/classify` |
|
||||
| `Rerank` | `/v1/rerank` |
|
||||
|
||||
The OpenAI RPCs carry the HTTP route's JSON body, so they are prepared exactly as
|
||||
over HTTP. A typed RPC is prepared from its fields with the same rules, and the
|
||||
router's additions (rid, PD bootstrap, DP rank, sampling defaults) are set on the
|
||||
proto it forwards. A text prompt is tokenized once, as its HTTP route forwards
|
||||
`input_ids`: `TextGenerate` is sent as `Generate` with `return_text`, so the engine
|
||||
still supplies the text, and `TextEmbed` as `Embed`. They stay text RPCs under
|
||||
`--disable-input-ids-forwarding`, without a tokenizer, or, for `TextGenerate`, with
|
||||
`return_text_in_logprobs`. Other RPCs, such as
|
||||
`Tokenize`, `HealthCheck` and the admin RPCs, answer `UNIMPLEMENTED`. Request metadata and `trace_headers`
|
||||
act as HTTP headers for routing and forwarding, and a cancelled call aborts the
|
||||
engine request. Router errors arrive as a gRPC status carrying the HTTP path's
|
||||
message and an `x-router-error-code` trailer; an engine's status passes through
|
||||
unchanged. An engine without `--grpc-port` (including one launched with
|
||||
`--api-key`, which native gRPC rejects) cannot serve gRPC calls: the router warns
|
||||
when it registers, and calls routed to it fail with `UNAVAILABLE`.
|
||||
|
||||
## Upgrading from `cache_aware_zmq`
|
||||
|
||||
The `cache_aware_zmq` policy has been removed. Configurations using it should
|
||||
select `--policy cache_aware` and choose a native cache-prefix source: the
|
||||
Router-local radix tree (the default), or the external Indexer shown above.
|
||||
|
||||
The legacy `--cache-threshold`, `--balance-abs-threshold`, and
|
||||
`--balance-rel-threshold` flags have also been removed. They do not have
|
||||
one-to-one replacements; remove them and review the current `sgl-router
|
||||
--help` output when tuning Cache-Aware routing.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
@@ -100,7 +100,7 @@ pub struct ModelArgs {
|
||||
/// Accepts temperature, top_p, top_k, min_p, repetition_penalty,
|
||||
/// frequency_penalty, presence_penalty, and n. Numeric values fill absent
|
||||
/// fields; {"min": LO, "max": HI} bands only constrain supplied values
|
||||
/// and require reject mode. See README for domains and null handling.
|
||||
/// and require reject mode. See REFERENCE.md for domains and null handling.
|
||||
#[arg(long, value_name = "JSON")]
|
||||
pub override_sampling_params: Option<String>,
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// SPDX-FileCopyrightText: Copyright (c) 2026 The SGLang Authors
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! Side-by-side implementation of POLICY_DESIGN.md. Chat routing can opt into
|
||||
//! Routing policies described in ARCHITECTURE.md. Chat routing can opt into
|
||||
//! this interface through AppContext; `policies` remains the default.
|
||||
|
||||
pub mod admission;
|
||||
|
||||
Reference in New Issue
Block a user