[sgl-router] Fix reorg admission proxy test build after BucketResolver::new (#40537)
Co-authored-by: Kan Wu <wukanustc@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Kan Wu
Claude Fable 5.1
parent
8d08dfdab7
commit
630b1ef322
@@ -303,21 +303,23 @@ Session assignments are scoped by model, bucket ID, stage, and session key.
|
||||
`SessionAwarePolicy::new(store, engine_load)` receives shared state; the caller
|
||||
owns the store's idle timeout and eviction task. Missing or empty session keys
|
||||
use power-of-two without creating assignments. A new or out-of-group binding
|
||||
uses power-of-two with `AllowAll`, then the session policy checks its selected
|
||||
engine before binding. A concurrent live assignment wins, but is checked before
|
||||
returning it; rejection ends that attempt without rewriting the binding or
|
||||
retrying another engine. Existing bindings are reused regardless of pressure
|
||||
when admitted. Session policies can be attached independently to each role.
|
||||
Programmatic reorg callers configure `model.affinity.session_id_header` for HTTP
|
||||
header extraction; this does not enable legacy global modes or backup escape.
|
||||
uses power-of-two with `AdmissionLimits::default()`, then the session policy
|
||||
checks its selected engine before binding. A concurrent live assignment wins,
|
||||
but is checked before returning it; rejection ends that attempt without
|
||||
rewriting the binding or retrying another engine. Existing bindings are reused
|
||||
regardless of pressure when admitted. Session policies can be attached
|
||||
independently to each role. Programmatic reorg callers configure
|
||||
`model.affinity.session_id_header` for HTTP header extraction; this does not
|
||||
enable legacy global modes or backup escape.
|
||||
|
||||
Session and sticky policies do not create assignments for missing keys. A
|
||||
binding outside the candidates cannot win. A missing binding may invoke policy
|
||||
fallback within the group; hard admission rejection remains an error.
|
||||
|
||||
Sticky fallback supports `round_robin`, `random`, `power_of_two`, and `load_based`,
|
||||
with round-robin as the default. Nested fallbacks use `AllowAll`; the owning
|
||||
policy explicitly checks the engine returned by its fallback.
|
||||
with round-robin as the default. Nested fallbacks use
|
||||
`AdmissionLimits::default()`; the owning policy explicitly checks the engine
|
||||
returned by its fallback.
|
||||
|
||||
### Cache-aware behavior
|
||||
|
||||
@@ -502,8 +504,8 @@ do not accept and ignore them.
|
||||
- Preserve configured capacity, pending-prefill, and in-flight checks, including
|
||||
their missing-report behavior. Power-of-two applies admission to its selected
|
||||
engine; other policies explicitly place checks in their selection logic.
|
||||
Other paths use `AllowAll` unless a check is configured. Never silently discard
|
||||
a configured budget.
|
||||
Other paths use `AdmissionLimits::default()` unless a check is configured.
|
||||
Never silently discard a configured budget.
|
||||
|
||||
Listener and shutdown configuration, discovery, worker health and circuit
|
||||
breakers, tokenizer loading, request timeouts, sampling overrides, and logging
|
||||
|
||||
@@ -87,8 +87,9 @@ impl Policy for SessionAwarePolicy {
|
||||
});
|
||||
}
|
||||
|
||||
// The nested power-of-two policy uses AllowAll. The session owner
|
||||
// checks its chosen engine before creating or replacing a binding.
|
||||
// The nested power-of-two policy uses AdmissionLimits::default().
|
||||
// The session owner checks its chosen engine before creating or
|
||||
// replacing a binding.
|
||||
let mut pick = self.pick_fallback(engines, request).await?;
|
||||
let load = self.engine_load.capture_snapshot(Instant::now());
|
||||
self.check(&pick.engine, &load)?;
|
||||
|
||||
@@ -169,7 +169,8 @@ async fn configured_limits_reject_before_dispatch_and_admit_after_load_drops() {
|
||||
worker_ids: Some([WorkerId("w".into())].into()),
|
||||
policy: Arc::new(policy),
|
||||
}),
|
||||
)]),
|
||||
)])
|
||||
.unwrap(),
|
||||
)]
|
||||
.into(),
|
||||
);
|
||||
|
||||
Reference in New Issue
Block a user