[sgl-router] Fix reorg admission proxy test build after BucketResolver::new (#40537)
Co-authored-by: Kan Wu <wukanustc@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Kan Wu
Claude Fable 5.1
parent
8d08dfdab7
commit
630b1ef322
@@ -303,21 +303,23 @@ Session assignments are scoped by model, bucket ID, stage, and session key.
|
|||||||
`SessionAwarePolicy::new(store, engine_load)` receives shared state; the caller
|
`SessionAwarePolicy::new(store, engine_load)` receives shared state; the caller
|
||||||
owns the store's idle timeout and eviction task. Missing or empty session keys
|
owns the store's idle timeout and eviction task. Missing or empty session keys
|
||||||
use power-of-two without creating assignments. A new or out-of-group binding
|
use power-of-two without creating assignments. A new or out-of-group binding
|
||||||
uses power-of-two with `AllowAll`, then the session policy checks its selected
|
uses power-of-two with `AdmissionLimits::default()`, then the session policy
|
||||||
engine before binding. A concurrent live assignment wins, but is checked before
|
checks its selected engine before binding. A concurrent live assignment wins,
|
||||||
returning it; rejection ends that attempt without rewriting the binding or
|
but is checked before returning it; rejection ends that attempt without
|
||||||
retrying another engine. Existing bindings are reused regardless of pressure
|
rewriting the binding or retrying another engine. Existing bindings are reused
|
||||||
when admitted. Session policies can be attached independently to each role.
|
regardless of pressure when admitted. Session policies can be attached
|
||||||
Programmatic reorg callers configure `model.affinity.session_id_header` for HTTP
|
independently to each role. Programmatic reorg callers configure
|
||||||
header extraction; this does not enable legacy global modes or backup escape.
|
`model.affinity.session_id_header` for HTTP header extraction; this does not
|
||||||
|
enable legacy global modes or backup escape.
|
||||||
|
|
||||||
Session and sticky policies do not create assignments for missing keys. A
|
Session and sticky policies do not create assignments for missing keys. A
|
||||||
binding outside the candidates cannot win. A missing binding may invoke policy
|
binding outside the candidates cannot win. A missing binding may invoke policy
|
||||||
fallback within the group; hard admission rejection remains an error.
|
fallback within the group; hard admission rejection remains an error.
|
||||||
|
|
||||||
Sticky fallback supports `round_robin`, `random`, `power_of_two`, and `load_based`,
|
Sticky fallback supports `round_robin`, `random`, `power_of_two`, and `load_based`,
|
||||||
with round-robin as the default. Nested fallbacks use `AllowAll`; the owning
|
with round-robin as the default. Nested fallbacks use
|
||||||
policy explicitly checks the engine returned by its fallback.
|
`AdmissionLimits::default()`; the owning policy explicitly checks the engine
|
||||||
|
returned by its fallback.
|
||||||
|
|
||||||
### Cache-aware behavior
|
### Cache-aware behavior
|
||||||
|
|
||||||
@@ -502,8 +504,8 @@ do not accept and ignore them.
|
|||||||
- Preserve configured capacity, pending-prefill, and in-flight checks, including
|
- Preserve configured capacity, pending-prefill, and in-flight checks, including
|
||||||
their missing-report behavior. Power-of-two applies admission to its selected
|
their missing-report behavior. Power-of-two applies admission to its selected
|
||||||
engine; other policies explicitly place checks in their selection logic.
|
engine; other policies explicitly place checks in their selection logic.
|
||||||
Other paths use `AllowAll` unless a check is configured. Never silently discard
|
Other paths use `AdmissionLimits::default()` unless a check is configured.
|
||||||
a configured budget.
|
Never silently discard a configured budget.
|
||||||
|
|
||||||
Listener and shutdown configuration, discovery, worker health and circuit
|
Listener and shutdown configuration, discovery, worker health and circuit
|
||||||
breakers, tokenizer loading, request timeouts, sampling overrides, and logging
|
breakers, tokenizer loading, request timeouts, sampling overrides, and logging
|
||||||
|
|||||||
@@ -87,8 +87,9 @@ impl Policy for SessionAwarePolicy {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// The nested power-of-two policy uses AllowAll. The session owner
|
// The nested power-of-two policy uses AdmissionLimits::default().
|
||||||
// checks its chosen engine before creating or replacing a binding.
|
// The session owner checks its chosen engine before creating or
|
||||||
|
// replacing a binding.
|
||||||
let mut pick = self.pick_fallback(engines, request).await?;
|
let mut pick = self.pick_fallback(engines, request).await?;
|
||||||
let load = self.engine_load.capture_snapshot(Instant::now());
|
let load = self.engine_load.capture_snapshot(Instant::now());
|
||||||
self.check(&pick.engine, &load)?;
|
self.check(&pick.engine, &load)?;
|
||||||
|
|||||||
@@ -169,7 +169,8 @@ async fn configured_limits_reject_before_dispatch_and_admit_after_load_drops() {
|
|||||||
worker_ids: Some([WorkerId("w".into())].into()),
|
worker_ids: Some([WorkerId("w".into())].into()),
|
||||||
policy: Arc::new(policy),
|
policy: Arc::new(policy),
|
||||||
}),
|
}),
|
||||||
)]),
|
)])
|
||||||
|
.unwrap(),
|
||||||
)]
|
)]
|
||||||
.into(),
|
.into(),
|
||||||
);
|
);
|
||||||
|
|||||||
Reference in New Issue
Block a user