minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[misc] Use --cuda-graph-max-bs-decode in tests, examples, and docs (#29591)

Browse Source
This commit is contained in:
Liangsheng Yin
2026-06-28 18:38:28 -07:00
committed by GitHub
parent 3217410cf6
commit 909123ddb8
140 changed files with 304 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5.2_examples.mdx
+2 -2
View File
@@ -245,7 +245,7 @@ do
--mem-fraction-static 0.8 \
--port 8000 \
--served-model-name glm-5 \
--cuda-graph-max-bs 32 \
--cuda-graph-max-bs-decode 32 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--speculative-draft-model-quantization unquant \
@@ -352,7 +352,7 @@ do
--port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --ep-size 16 \
--mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \
--served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \
--cuda-graph-max-bs 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
--cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
--tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \
--speculative-draft-model-quantization unquant \
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx
+1 -1
View File
@@ -238,7 +238,7 @@ These arguments and environment variables are critical for tuning decode perform
<td>Enabled</td>
</tr>
<tr>
<td>`--cuda-graph-max-bs`</td>
<td>`--cuda-graph-max-bs-decode`</td>
<td>Caps the maximum batch size for which CUDA Graphs are captured. Larger values cover more batch sizes but increase graph capture time and memory overhead. If your `max-running-requests` is high but typical batch sizes are lower, use a smaller value to reduce capture overhead. Tune based on your observed batch size distribution during serving.</td>
<td>`4`</td>
</tr>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx
+1 -1
View File
@@ -2072,7 +2072,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs-decode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_v3_2.mdx
+3 -3
View File
@@ -168,7 +168,7 @@ do
--chunked-prefill-size -1 \
--max-prefill-tokens 68000 \
--max-running-requests 32 \
--cuda-graph-max-bs 4 \
--cuda-graph-max-bs-decode 4 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--quantization modelslim \
@@ -363,7 +363,7 @@ do
--chunked-prefill-size -1 \
--max-prefill-tokens 68000 \
--max-running-requests 32 \
--cuda-graph-max-bs 4 \
--cuda-graph-max-bs-decode 4 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--quantization modelslim \
@@ -558,7 +558,7 @@ do
--chunked-prefill-size -1 \
--max-prefill-tokens 68000 \
--max-running-requests 32 \
--cuda-graph-max-bs 4 \
--cuda-graph-max-bs-decode 4 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--quantization modelslim \
docs_new/docs/hardware-platforms/ascend-npus/best_practice/glm5_1.mdx
+1 -1
View File
@@ -112,7 +112,7 @@ do
--trust-remote-code \
--mem-fraction-static 0.65 \
--served-model-name glm-5 \
--cuda-graph-max-bs 16 \
--cuda-graph-max-bs-decode 16 \
--max-running-requests 256 \
--quantization modelslim \
--speculative-draft-model-quantization unquant \
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ v0.5.13 or a later version.
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 4 8 20 21 22` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 4 8 20 21 22` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ recommended to use v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 3 4 5 6 7 8` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 3 4 5 6 7 8` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx
+1 -1
View File
@@ -31,7 +31,7 @@ v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 3 4 5 6` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 3 4 5 6` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx
+1 -1
View File
@@ -29,7 +29,7 @@ v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 4 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 5 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx
+1 -1
View File
@@ -29,7 +29,7 @@ v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 8 16 24 32 48 64 80` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 8 16 24 32 48 64 80` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ version.
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 6 9 10 15 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 6 9 10 15 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 94208` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.13 or a later version.
| Tensor Parallelism | `--tp-size 2` |
| Data Parallelism | `--dp-size 2` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx
+1 -1
View File
@@ -24,7 +24,7 @@ version.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 4` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 4 8 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 4 8 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx
+1 -1
View File
@@ -28,7 +28,7 @@ use v0.5.13 or a later version.
| Data Parallelism | `--dp-size 8` |
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.13 or a later version.
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 2 8 16 32 48` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 8 16 32 48` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.13 or a later version.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 2` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ v0.5.13 or a later version.
| Data Parallelism | `--dp-size 2` |
| Expert Parallelism | `--ep-size 4 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 2 4 8` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 4 8` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |