Files
sglang/python
minke.yu c401c614fa fix(dp): rank uncached candidates by published backlog only
Deriving each candidate's uncached estimate from its own recent hit rate is
self-reinforcing: a rank that falls behind sees its hit rate drop (fewer
chances to hit), gets charged more for every new request, and falls further
behind. Measured on c580: one rank ended 16% below the others, 30 s cross-rank
dispersion max 952x, and 9.6% lower rps than the total_tokens arm.

Charge the fleet-average uncached size instead and rank by the published
num_waiting_uncached_tokens, which already carries each rank's exact prefix
match.
2026-11-11 02:06:59 +08:00
..