feat(dp): uncached_tokens load-balance method

DP balancing on total_tokens charges each request's whole prompt, but with
prefix caching only the unmatched part costs prefill compute. On a workload
with a 95% hit rate the prompt is ~95% noise: per request the uncached part
is a few percent of the prompt yet varies by more than an order of magnitude
(p50 1.9k / p99 59k tokens against a 95k prompt), so one cold request
outweighs many hot ones and per-rank prefill work drifts apart.

Add --load-balance-method uncached_tokens: DPBudget tracks the published
LoadSnapshot.num_waiting_uncached_tokens (prefix match minus accepted storage
hits) plus the rank's recent cache_hit_rate for a newly dispatched request.
This commit is contained in:
2026-11-11 00:03:55 +08:00
parent fef023b64f
commit 24b284d4e7
4 changed files with 1302 additions and 1190 deletions
@@ -98,6 +98,7 @@ class Parallel(msgspec.Struct):
"follow_bootstrap_room",
"total_requests",
"total_tokens",
"uncached_tokens",
],
),
] = "auto"
File diff suppressed because it is too large Load Diff