[Refactor] Cuda Graph Runner/Backend Refactor (#23906)
Co-authored-by: BBuf <1182563586@qq.com> Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com> Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
This commit is contained in:
co-authored by
BBuf
Cheng Wan
Lianmin Zheng
parent
56f06278c6
commit
2495c02c2c
@@ -325,7 +325,7 @@ def _run_server_and_generate(
|
||||
"--mem-fraction-static",
|
||||
"0.5",
|
||||
"--disable-cuda-graph",
|
||||
"--disable-piecewise-cuda-graph",
|
||||
"--cuda-graph-backend-prefill=disabled",
|
||||
"--disable-radix-cache",
|
||||
]
|
||||
if extra_server_args:
|
||||
|
||||
Reference in New Issue
Block a user