[Performance] Decode Offload improves the long texts performance 100% through dynamic block offload. (#17216)

Co-authored-by: zhangheng <hzh0425@apache.org>
This commit is contained in:
yuyu5333
2026-03-08 17:16:53 +08:00
committed by GitHub
co-authored by zhangheng
parent 97a2a9be0f
commit 230fb55899
8 changed files with 290 additions and 33 deletions
@@ -576,6 +576,8 @@ class HiCacheController:
model_name: Optional[str] = None,
storage_backend_extra_config: Optional[dict] = None,
):
if storage_backend_extra_config is None:
storage_backend_extra_config = {}
if is_dp_attention_enabled():
self.tp_rank = get_attention_tp_rank()