[misc] Use --cuda-graph-max-bs-decode in tests, examples, and docs (#29591)

This commit is contained in:
Liangsheng Yin
2026-06-28 18:38:28 -07:00
committed by GitHub
parent 3217410cf6
commit 909123ddb8
140 changed files with 304 additions and 290 deletions
@@ -64,10 +64,10 @@ If you encounter out-of-memory (OOM) errors, you can adjust the following parame
- If OOM occurs during decoding, try lowering `--max-running-requests`.
- You can also reduce `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput.
### Tune `--cuda-graph-max-bs`
### Tune `--cuda-graph-max-bs-decode`
By default, CUDA graph is enabled only for small batch sizes (e.g., less than 160 or 256).
However, for some models, especially at large tensor parallelism sizes, CUDA graph can be useful for batch sizes up to 512 or 768.
Therefore, it may be beneficial to increase `--cuda-graph-max-bs` to a larger value.
Therefore, it may be beneficial to increase `--cuda-graph-max-bs-decode` to a larger value.
Note that CUDA graph consumes more memory, so you may need to reduce `--mem-fraction-static` at the same time.
### Tune `--dp-size` and `--tp-size`