mem_cache: allow disabling the DSv4.1 SWA prefix-match gate via env
The unified-cache SWA component gates prefix matching on a live SWA window, so a long prefix only matches up to the last sliding window and the tail is recomputed. Add SGLANG_DSV4_SWA_GATES_PREFIX_MATCH (default True) so the gate can be turned off for measurement; the SWA window is then rebuilt/recovered downstream as for the req-ring paths. Measured on 4xB300 DP4 at 160k prefix: c32 hit tokens 40.6% -> ~100% (with HiCache L2/L3 enabled), needle test (cold vs hit) identical.
This commit is contained in:
@@ -633,6 +633,10 @@ class Envs:
|
||||
# size the KV pool after CUDA-graph capture
|
||||
SGLANG_ENABLE_POST_CAPTURE_KV_SIZING = EnvBool(False)
|
||||
|
||||
# DSv4.1 unified cache: allow a FULL prefix match even when the SWA component has
|
||||
# no live value (the SWA window is rebuilt/recovered downstream).
|
||||
SGLANG_DSV4_SWA_GATES_PREFIX_MATCH = EnvBool(True)
|
||||
|
||||
# ===================================================================
|
||||
# Scheduler token budgeting and admission
|
||||
# ===================================================================
|
||||
|
||||
@@ -352,7 +352,7 @@ def build_kv_cache(
|
||||
# SWA the tree does not own must not gate prefix matching: a per-request ring,
|
||||
# or paged SWA beside the target's encoder-replay request window, which then
|
||||
# only the draft uses.
|
||||
swa_gates_prefix_match = not (
|
||||
swa_gates_prefix_match = envs.SGLANG_DSV4_SWA_GATES_PREFIX_MATCH.get() and not (
|
||||
is_swa_req_ring(token_to_kv_pool_allocator)
|
||||
or (
|
||||
isinstance(token_to_kv_pool, DeepSeekV4TokenToKVPool)
|
||||
|
||||
Reference in New Issue
Block a user