minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[misc] Use --cuda-graph-max-bs-decode in tests, examples, and docs (#29591)

Browse Source
This commit is contained in:
Liangsheng Yin
2026-06-28 18:38:28 -07:00
committed by GitHub
parent 3217410cf6
commit 909123ddb8
140 changed files with 304 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/advanced_features/hyperparameter_tuning.mdx
+2 -2
View File
@@ -64,10 +64,10 @@ If you encounter out-of-memory (OOM) errors, you can adjust the following parame
- If OOM occurs during decoding, try lowering `--max-running-requests`.
- You can also reduce `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput.
### Tune `--cuda-graph-max-bs`
### Tune `--cuda-graph-max-bs-decode`
By default, CUDA graph is enabled only for small batch sizes (e.g., less than 160 or 256).
However, for some models, especially at large tensor parallelism sizes, CUDA graph can be useful for batch sizes up to 512 or 768.
Therefore, it may be beneficial to increase `--cuda-graph-max-bs` to a larger value.
Therefore, it may be beneficial to increase `--cuda-graph-max-bs-decode` to a larger value.
Note that CUDA graph consumes more memory, so you may need to reduce `--mem-fraction-static` at the same time.
### Tune `--dp-size` and `--tp-size`
docs_new/docs/advanced_features/lora.ipynb
+2 -2
View File
@@ -228,7 +228,7 @@
"server_process, port = launch_server_cmd(\"\"\"\n",
" python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
" --enable-lora \\\n",
" --cuda-graph-max-bs 2 \\\n",
" --cuda-graph-max-bs-decode 2 \\\n",
" --max-loras-per-batch 2 \\\n",
" --max-lora-rank 256\n",
" --lora-target-modules all\n",
@@ -434,7 +434,7 @@
"server_process, port = launch_server_cmd(\"\"\"\n",
" python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
" --enable-lora \\\n",
" --cuda-graph-max-bs 8 \\\n",
" --cuda-graph-max-bs-decode 8 \\\n",
" --max-loras-per-batch 3 \\\n",
" --max-lora-rank 256 \\\n",
" --lora-target-modules all \\\n",
docs_new/docs/advanced_features/lora.mdx
+2 -2
View File
@@ -165,7 +165,7 @@ server_process, port = launch_server_cmd(
"""
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--enable-lora \
--cuda-graph-max-bs 2 \
--cuda-graph-max-bs-decode 2 \
--max-loras-per-batch 2 \
--max-lora-rank 256
--lora-target-modules all
@@ -314,7 +314,7 @@ server_process, port = launch_server_cmd(
"""
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--enable-lora \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--max-loras-per-batch 3 \
--max-lora-rank 256 \
--lora-target-modules all \
docs_new/docs/advanced_features/server_arguments.mdx
+1 -1
View File
@@ -2489,7 +2489,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs-decode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-max-bs-decode</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
docs_new/docs/advanced_features/speculative_decoding.ipynb
+4 -4
View File
@@ -36,7 +36,7 @@
"\n",
"You can find the best combinations of these parameters with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py).\n",
"\n",
"In the documentation below, we set `--cuda-graph-max-bs` to be a small value for faster engine startup. For your own workloads, please tune the above parameters together with `--cuda-graph-max-bs`, `--max-running-requests`, `--mem-fraction-static` for the best performance. "
"In the documentation below, we set `--cuda-graph-max-bs-decode` to be a small value for faster engine startup. For your own workloads, please tune the above parameters together with `--cuda-graph-max-bs-decode`, `--max-running-requests`, `--mem-fraction-static` for the best performance. "
]
},
{
@@ -69,7 +69,7 @@
"server_process, port = launch_server_cmd(\"\"\"\n",
"python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf --speculative-algorithm EAGLE \\\n",
" --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 3 \\\n",
" --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 --cuda-graph-max-bs 8 --log-level warning\n",
" --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 --cuda-graph-max-bs-decode 8 --log-level warning\n",
"\"\"\")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
@@ -181,7 +181,7 @@
"python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3-8B-Instruct --speculative-algorithm EAGLE \\\n",
" --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \\\n",
" --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \\\n",
" --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16 --log-level warning\n",
" --mem-fraction 0.7 --cuda-graph-max-bs-decode 2 --dtype float16 --log-level warning\n",
"\"\"\")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
@@ -235,7 +235,7 @@
"python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algorithm EAGLE3 \\\n",
" --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B --speculative-num-steps 5 \\\n",
" --speculative-eagle-topk 8 --speculative-num-draft-tokens 32 --mem-fraction 0.6 \\\n",
" --cuda-graph-max-bs 2 --dtype float16 --log-level warning\n",
" --cuda-graph-max-bs-decode 2 --dtype float16 --log-level warning\n",
"\"\"\")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
docs_new/docs/advanced_features/speculative_decoding.mdx
+11 -11
View File
@@ -242,7 +242,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 4 \
--speculative-num-draft-tokens 16 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -326,7 +326,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 16 \
--speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--dtype float16 \
--log-level warning
```
@@ -365,7 +365,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 4 \
--speculative-num-draft-tokens 16 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--dtype float16 \
--log-level warning
```
@@ -405,7 +405,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -557,7 +557,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 2 \
--speculative-num-draft-tokens 7 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -600,7 +600,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 5 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -695,7 +695,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 16 \
--speculative-ngram-max-bfs-breadth 10 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -972,10 +972,10 @@ Below is a comprehensive list of all speculative decoding parameters available i
```bash Command
# Fewer CUDA graph captures = less memory reserved
--cuda-graph-max-bs 4 # or even 2 for tight memory situations
--cuda-graph-max-bs-decode 4 # or even 2 for tight memory situations
```
- If omitted, `--cuda-graph-max-bs` is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs.
- If omitted, `--cuda-graph-max-bs-decode` is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs.
### Step 3: Reduce draft tree size
@@ -1008,13 +1008,13 @@ python3 -m sglang.launch_server \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--cuda-graph-max-bs 2 \
--cuda-graph-max-bs-decode 2 \
--mem-fraction-static 0.5 \
--max-running-requests 4 \
--log-level warning
```
Then gradually increase `--speculative-num-draft-tokens`, `--speculative-eagle-topk`, and `--cuda-graph-max-bs`. Increase `--mem-fraction-static` last, only after the run is stable.
Then gradually increase `--speculative-num-draft-tokens`, `--speculative-eagle-topk`, and `--cuda-graph-max-bs-decode`. Increase `--mem-fraction-static` last, only after the run is stable.
---