feat(dp): uncached_tokens load-balance method
DP balancing on total_tokens charges each request's whole prompt, but with prefix caching only the unmatched part costs prefill compute. On a workload with a 95% hit rate the prompt is ~95% noise: per request the uncached part is a few percent of the prompt yet varies by more than an order of magnitude (p50 1.9k / p99 59k tokens against a 95k prompt), so one cold request outweighs many hot ones and per-rank prefill work drifts apart. Add --load-balance-method uncached_tokens: DPBudget tracks the published LoadSnapshot.num_waiting_uncached_tokens (prefix match minus accepted storage hits) plus the rank's recent cache_hit_rate for a newly dispatched request.
This commit is contained in:
@@ -98,6 +98,7 @@ class Parallel(msgspec.Struct):
|
||||
"follow_bootstrap_room",
|
||||
"total_requests",
|
||||
"total_tokens",
|
||||
"uncached_tokens",
|
||||
],
|
||||
),
|
||||
] = "auto"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user