[HiCache] Merge HiCache event checks to reduce decode overhead (#30511)
Co-authored-by: alphabetc1 <47200617+alphabetc1@users.noreply.github.com>
This commit is contained in:
co-authored by
alphabetc1
parent
bfc450248e
commit
50029f05a3
@@ -3226,11 +3226,6 @@ class Scheduler(
|
||||
batch.batch_is_full = False
|
||||
return batch
|
||||
|
||||
# Eagerly release lock_ref on completed write-through nodes so they
|
||||
# become evictable, improving batch scheduling headroom.
|
||||
if self.enable_hierarchical_cache:
|
||||
self.tree_cache.flush_write_through_acks()
|
||||
|
||||
# Check if decode out of memory
|
||||
if (kv_full_retract_flag := not batch.check_decode_mem()) or (
|
||||
TEST_RETRACT and self.forward_ct % TEST_RETRACT_INTERVAL == 0
|
||||
|
||||
Reference in New Issue
Block a user