Support batch size > 1 when enable CP (#23269)
Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Co-authored-by: Khoa Pham <khoa.pham@radixark.ai> Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
This commit is contained in:
co-authored by
Shunkang
Khoa Pham
Baizhou Zhang
parent
ddf0627254
commit
19663aafcd
@@ -826,9 +826,7 @@ class PrefillAdder:
|
||||
# TODO support cp with multiple requests
|
||||
# Enabling context parallelism currently presents precision issues;
|
||||
# therefore, the prefill-batch setting is temporarily set to 1.
|
||||
if (
|
||||
self.dsa_prefill_cp_in_seq_split or self.prefill_context_parallel_enabled
|
||||
) and len(self.can_run_list) >= 1:
|
||||
if (self.dsa_prefill_cp_in_seq_split) and len(self.can_run_list) >= 1:
|
||||
return AddReqResult.OTHER
|
||||
|
||||
if (x := self.prefill_max_requests) is not None and len(self.can_run_list) >= x:
|
||||
|
||||
Reference in New Issue
Block a user