Support batch size > 1 when enable CP (#23269)

Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
This commit is contained in:
Shunkangz
2026-05-27 14:11:17 -07:00
committed by GitHub
co-authored by Shunkang Khoa Pham Baizhou Zhang
parent ddf0627254
commit 19663aafcd
13 changed files with 263 additions and 300 deletions
@@ -826,9 +826,7 @@ class PrefillAdder:
# TODO support cp with multiple requests
# Enabling context parallelism currently presents precision issues;
# therefore, the prefill-batch setting is temporarily set to 1.
if (
self.dsa_prefill_cp_in_seq_split or self.prefill_context_parallel_enabled
) and len(self.can_run_list) >= 1:
if (self.dsa_prefill_cp_in_seq_split) and len(self.can_run_list) >= 1:
return AddReqResult.OTHER
if (x := self.prefill_max_requests) is not None and len(self.can_run_list) >= x: