minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[misc] Use --cuda-graph-max-bs-decode in tests, examples, and docs (#29591)

Browse Source
This commit is contained in:
Liangsheng Yin
2026-06-28 18:38:28 -07:00
committed by GitHub
parent 3217410cf6
commit 909123ddb8
140 changed files with 304 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmark/deepseek_v3/README.md
+3 -3
View File
@@ -177,19 +177,19 @@ There is one B200 node with 4 (for FP4) GPUs or 8 (for FP4 or FP8) GPUs. Both F
If using 4 GPUs:
```bash
python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-0528-FP4-V2 --host 0.0.0.0 --port 8000 --tensor-parallel-size=4 --cuda-graph-max-bs 256 --max-running-requests 256 --mem-fraction-static 0.85 --ep-size 4 --scheduler-recv-interval 30 --enable-symm-mem --stream-interval 10
python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-0528-FP4-V2 --host 0.0.0.0 --port 8000 --tensor-parallel-size=4 --cuda-graph-max-bs-decode 256 --max-running-requests 256 --mem-fraction-static 0.85 --ep-size 4 --scheduler-recv-interval 30 --enable-symm-mem --stream-interval 10
```
If using 8 GPUs:
```bash
python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-0528-FP4-V2 --host 0.0.0.0 --port 8000 --tensor-parallel-size=8 --cuda-graph-max-bs 256 --max-running-requests 256 --mem-fraction-static 0.85 --ep-size 8 --scheduler-recv-interval 30 --enable-symm-mem --stream-interval 10
python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-0528-FP4-V2 --host 0.0.0.0 --port 8000 --tensor-parallel-size=8 --cuda-graph-max-bs-decode 256 --max-running-requests 256 --mem-fraction-static 0.85 --ep-size 8 --scheduler-recv-interval 30 --enable-symm-mem --stream-interval 10
```
#### FP8
```bash
SGLANG_ENABLE_JIT_DEEPGEMM=false python3 -m sglang.launch_server --model-path=deepseek-ai/DeepSeek-R1-0528 --host=0.0.0.0 --port=8000 --tensor-parallel-size=8 --cuda-graph-max-bs 128 --max-running-requests 128 --mem-fraction-static 0.82 --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 32768 --max-prefill-tokens 32768 --scheduler-recv-interval 30 --stream-interval 30 --fp8-gemm-backend flashinfer_trtllm
SGLANG_ENABLE_JIT_DEEPGEMM=false python3 -m sglang.launch_server --model-path=deepseek-ai/DeepSeek-R1-0528 --host=0.0.0.0 --port=8000 --tensor-parallel-size=8 --cuda-graph-max-bs-decode 128 --max-running-requests 128 --mem-fraction-static 0.82 --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 32768 --max-prefill-tokens 32768 --scheduler-recv-interval 30 --stream-interval 30 --fp8-gemm-backend flashinfer_trtllm
```
### Example: Serving with two H200\*8 nodes and docker
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx
+1 -1
View File
@@ -497,7 +497,7 @@ sglang_args=$(echo serve \
--dsa-decode-backend flashmla_sparse \
--enable-dsa-prefill-context-parallel \
--dsa-prefill-cp-mode round-robin-split \
--cuda-graph-max-bs 128 \
--cuda-graph-max-bs-decode 128 \
--max-running-requests 128 \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+1 -1
View File
@@ -174,7 +174,7 @@ Both Instruct repos ship as **FP4 MoE experts + FP8 attention / dense** (one mix
**Concurrency & DeepEP dispatch buffer**
Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs`, `--max-running-requests`, and the env together.
Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs-decode`, `--max-running-requests`, and the env together.
The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised.
docs_new/docs/advanced_features/hyperparameter_tuning.mdx
+2 -2
View File
@@ -64,10 +64,10 @@ If you encounter out-of-memory (OOM) errors, you can adjust the following parame
- If OOM occurs during decoding, try lowering `--max-running-requests`.
- You can also reduce `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput.
### Tune `--cuda-graph-max-bs`
### Tune `--cuda-graph-max-bs-decode`
By default, CUDA graph is enabled only for small batch sizes (e.g., less than 160 or 256).
However, for some models, especially at large tensor parallelism sizes, CUDA graph can be useful for batch sizes up to 512 or 768.
Therefore, it may be beneficial to increase `--cuda-graph-max-bs` to a larger value.
Therefore, it may be beneficial to increase `--cuda-graph-max-bs-decode` to a larger value.
Note that CUDA graph consumes more memory, so you may need to reduce `--mem-fraction-static` at the same time.
### Tune `--dp-size` and `--tp-size`
docs_new/docs/advanced_features/lora.ipynb
+2 -2
View File
@@ -228,7 +228,7 @@
"server_process, port = launch_server_cmd(\"\"\"\n",
" python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
" --enable-lora \\\n",
" --cuda-graph-max-bs 2 \\\n",
" --cuda-graph-max-bs-decode 2 \\\n",
" --max-loras-per-batch 2 \\\n",
" --max-lora-rank 256\n",
" --lora-target-modules all\n",
@@ -434,7 +434,7 @@
"server_process, port = launch_server_cmd(\"\"\"\n",
" python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
" --enable-lora \\\n",
" --cuda-graph-max-bs 8 \\\n",
" --cuda-graph-max-bs-decode 8 \\\n",
" --max-loras-per-batch 3 \\\n",
" --max-lora-rank 256 \\\n",
" --lora-target-modules all \\\n",
docs_new/docs/advanced_features/lora.mdx
+2 -2
View File
@@ -165,7 +165,7 @@ server_process, port = launch_server_cmd(
"""
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--enable-lora \
--cuda-graph-max-bs 2 \
--cuda-graph-max-bs-decode 2 \
--max-loras-per-batch 2 \
--max-lora-rank 256
--lora-target-modules all
@@ -314,7 +314,7 @@ server_process, port = launch_server_cmd(
"""
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--enable-lora \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--max-loras-per-batch 3 \
--max-lora-rank 256 \
--lora-target-modules all \
docs_new/docs/advanced_features/server_arguments.mdx
+1 -1
View File
@@ -2489,7 +2489,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs-decode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-max-bs-decode</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
docs_new/docs/advanced_features/speculative_decoding.ipynb
+4 -4
View File
@@ -36,7 +36,7 @@
"\n",
"You can find the best combinations of these parameters with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py).\n",
"\n",
"In the documentation below, we set `--cuda-graph-max-bs` to be a small value for faster engine startup. For your own workloads, please tune the above parameters together with `--cuda-graph-max-bs`, `--max-running-requests`, `--mem-fraction-static` for the best performance. "
"In the documentation below, we set `--cuda-graph-max-bs-decode` to be a small value for faster engine startup. For your own workloads, please tune the above parameters together with `--cuda-graph-max-bs-decode`, `--max-running-requests`, `--mem-fraction-static` for the best performance. "
]
},
{
@@ -69,7 +69,7 @@
"server_process, port = launch_server_cmd(\"\"\"\n",
"python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf --speculative-algorithm EAGLE \\\n",
" --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 3 \\\n",
" --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 --cuda-graph-max-bs 8 --log-level warning\n",
" --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 --cuda-graph-max-bs-decode 8 --log-level warning\n",
"\"\"\")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
@@ -181,7 +181,7 @@
"python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3-8B-Instruct --speculative-algorithm EAGLE \\\n",
" --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \\\n",
" --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \\\n",
" --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16 --log-level warning\n",
" --mem-fraction 0.7 --cuda-graph-max-bs-decode 2 --dtype float16 --log-level warning\n",
"\"\"\")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
@@ -235,7 +235,7 @@
"python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algorithm EAGLE3 \\\n",
" --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B --speculative-num-steps 5 \\\n",
" --speculative-eagle-topk 8 --speculative-num-draft-tokens 32 --mem-fraction 0.6 \\\n",
" --cuda-graph-max-bs 2 --dtype float16 --log-level warning\n",
" --cuda-graph-max-bs-decode 2 --dtype float16 --log-level warning\n",
"\"\"\")\n",
"\n",
"wait_for_server(f\"http://localhost:{port}\")"
docs_new/docs/advanced_features/speculative_decoding.mdx
+11 -11
View File
@@ -242,7 +242,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 4 \
--speculative-num-draft-tokens 16 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -326,7 +326,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 16 \
--speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--dtype float16 \
--log-level warning
```
@@ -365,7 +365,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 4 \
--speculative-num-draft-tokens 16 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--dtype float16 \
--log-level warning
```
@@ -405,7 +405,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -557,7 +557,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 2 \
--speculative-num-draft-tokens 7 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -600,7 +600,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 5 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -695,7 +695,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 16 \
--speculative-ngram-max-bfs-breadth 10 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--log-level warning
```
@@ -972,10 +972,10 @@ Below is a comprehensive list of all speculative decoding parameters available i
```bash Command
# Fewer CUDA graph captures = less memory reserved
--cuda-graph-max-bs 4 # or even 2 for tight memory situations
--cuda-graph-max-bs-decode 4 # or even 2 for tight memory situations
```
- If omitted, `--cuda-graph-max-bs` is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs.
- If omitted, `--cuda-graph-max-bs-decode` is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs.
### Step 3: Reduce draft tree size
@@ -1008,13 +1008,13 @@ python3 -m sglang.launch_server \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--cuda-graph-max-bs 2 \
--cuda-graph-max-bs-decode 2 \
--mem-fraction-static 0.5 \
--max-running-requests 4 \
--log-level warning
```
Then gradually increase `--speculative-num-draft-tokens`, `--speculative-eagle-topk`, and `--cuda-graph-max-bs`. Increase `--mem-fraction-static` last, only after the run is stable.
Then gradually increase `--speculative-num-draft-tokens`, `--speculative-eagle-topk`, and `--cuda-graph-max-bs-decode`. Increase `--mem-fraction-static` last, only after the run is stable.
---
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5.2_examples.mdx
+2 -2
View File
@@ -245,7 +245,7 @@ do
--mem-fraction-static 0.8 \
--port 8000 \
--served-model-name glm-5 \
--cuda-graph-max-bs 32 \
--cuda-graph-max-bs-decode 32 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--speculative-draft-model-quantization unquant \
@@ -352,7 +352,7 @@ do
--port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --ep-size 16 \
--mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \
--served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \
--cuda-graph-max-bs 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
--cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
--tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \
--speculative-draft-model-quantization unquant \
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx
+1 -1
View File
@@ -238,7 +238,7 @@ These arguments and environment variables are critical for tuning decode perform
<td>Enabled</td>
</tr>
<tr>
<td>`--cuda-graph-max-bs`</td>
<td>`--cuda-graph-max-bs-decode`</td>
<td>Caps the maximum batch size for which CUDA Graphs are captured. Larger values cover more batch sizes but increase graph capture time and memory overhead. If your `max-running-requests` is high but typical batch sizes are lower, use a smaller value to reduce capture overhead. Tune based on your observed batch size distribution during serving.</td>
<td>`4`</td>
</tr>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx
+1 -1
View File
@@ -2072,7 +2072,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs-decode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_v3_2.mdx
+3 -3
View File
@@ -168,7 +168,7 @@ do
--chunked-prefill-size -1 \
--max-prefill-tokens 68000 \
--max-running-requests 32 \
--cuda-graph-max-bs 4 \
--cuda-graph-max-bs-decode 4 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--quantization modelslim \
@@ -363,7 +363,7 @@ do
--chunked-prefill-size -1 \
--max-prefill-tokens 68000 \
--max-running-requests 32 \
--cuda-graph-max-bs 4 \
--cuda-graph-max-bs-decode 4 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--quantization modelslim \
@@ -558,7 +558,7 @@ do
--chunked-prefill-size -1 \
--max-prefill-tokens 68000 \
--max-running-requests 32 \
--cuda-graph-max-bs 4 \
--cuda-graph-max-bs-decode 4 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--quantization modelslim \
docs_new/docs/hardware-platforms/ascend-npus/best_practice/glm5_1.mdx
+1 -1
View File
@@ -112,7 +112,7 @@ do
--trust-remote-code \
--mem-fraction-static 0.65 \
--served-model-name glm-5 \
--cuda-graph-max-bs 16 \
--cuda-graph-max-bs-decode 16 \
--max-running-requests 256 \
--quantization modelslim \
--speculative-draft-model-quantization unquant \
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ v0.5.13 or a later version.
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 4 8 20 21 22` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 4 8 20 21 22` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ recommended to use v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 3 4 5 6 7 8` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 3 4 5 6 7 8` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx
+1 -1
View File
@@ -31,7 +31,7 @@ v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 3 4 5 6` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 3 4 5 6` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx
+1 -1
View File
@@ -29,7 +29,7 @@ v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 4 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 5 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx
+1 -1
View File
@@ -29,7 +29,7 @@ v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 8 16 24 32 48 64 80` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 8 16 24 32 48 64 80` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ version.
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 6 9 10 15 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 6 9 10 15 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 94208` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.13 or a later version.
| Tensor Parallelism | `--tp-size 2` |
| Data Parallelism | `--dp-size 2` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx
+1 -1
View File
@@ -24,7 +24,7 @@ version.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 4` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 4 8 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 4 8 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx
+1 -1
View File
@@ -28,7 +28,7 @@ use v0.5.13 or a later version.
| Data Parallelism | `--dp-size 8` |
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.13 or a later version.
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 2 8 16 32 48` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 8 16 32 48` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.13 or a later version.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 2` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ v0.5.13 or a later version.
| Data Parallelism | `--dp-size 2` |
| Expert Parallelism | `--ep-size 4 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 2 4 8` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 4 8` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs_new/docs/references/faq.mdx
+1 -1
View File
@@ -23,7 +23,7 @@ This error may result from kernel errors or out-of-memory issues:
### The server hangs
- If the server hangs during initialization or running, it can be memory issues (out of memory), network issues (nccl errors), or other bugs in sglang.
- If it is out of memory, you might see that `avail mem` is very low during the initialization or right after initialization. In this case,
you can try to decrease `--mem-fraction-static`, decrease `--cuda-graph-max-bs`, or decrease `--chunked-prefill-size`.
you can try to decrease `--mem-fraction-static`, decrease `--cuda-graph-max-bs-decode`, or decrease `--chunked-prefill-size`.
- Other bugs, please file an issue on GitHub.
docs_new/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy.mdx
+2 -2
View File
@@ -384,7 +384,7 @@ spec:
- "32768"
- --disaggregation-ib-device
- "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3"
- --cuda-graph-max-bs
- --cuda-graph-max-bs-decode
- "64"
- --max-running-requests
- "2048"
@@ -519,7 +519,7 @@ spec:
- --disaggregation-ib-device
# should modify according your rdma env
- "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3"
- --cuda-graph-max-bs
- --cuda-graph-max-bs-decode
- "64"
- --max-running-requests
- "2048"
docs_new/docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd.mdx
+2 -2
View File
@@ -285,7 +285,7 @@ spec:
- "2048"
- --tp-size
- "8" # Size of Tensor Parallelism
- --cuda-graph-max-bs
- --cuda-graph-max-bs-decode
- "16"
- --dist-init-addr
- $(LWS_LEADER_ADDRESS):20102
@@ -356,7 +356,7 @@ spec:
- mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
- --max-running-requests
- "4096"
- --cuda-graph-max-bs
- --cuda-graph-max-bs-decode
- "16"
- --tp-size
- "8" # Size of Tensor Parallelism
docs_new/src/snippets/autoregressive/deepseek-r1-advanced-deployment.jsx
+17 -17
View File
@@ -83,7 +83,7 @@ const lookupData = {
"parameters": {
"model_path": "nvidia/DeepSeek-R1-0528-FP4-v2",
"tensor_parallel_size": 4,
"cuda_graph_max_bs": 256,
"cuda_graph_max_bs_decode": 256,
"max_running_requests": 256,
"mem_fraction_static": 0.85,
"ep_size": 4,
@@ -100,7 +100,7 @@ const lookupData = {
"parameters": {
"model_path": "nvidia/DeepSeek-R1-0528-FP4-v2",
"tensor_parallel_size": 4,
"cuda_graph_max_bs": 256,
"cuda_graph_max_bs_decode": 256,
"max_running_requests": 256,
"mem_fraction_static": 0.85,
"ep_size": 4,
@@ -117,7 +117,7 @@ const lookupData = {
"parameters": {
"model_path": "nvidia/DeepSeek-R1-0528-FP4-v2",
"tensor_parallel_size": 8,
"cuda_graph_max_bs": 256,
"cuda_graph_max_bs_decode": 256,
"max_running_requests": 256,
"mem_fraction_static": 0.85,
"kv_cache_dtype": "fp8_e4m3",
@@ -136,7 +136,7 @@ const lookupData = {
"parameters": {
"model_path": "nvidia/DeepSeek-R1-0528-FP4-v2",
"tensor_parallel_size": 8,
"cuda_graph_max_bs": 256,
"cuda_graph_max_bs_decode": 256,
"max_running_requests": 256,
"mem_fraction_static": 0.85,
"kv_cache_dtype": "fp8_e4m3",
@@ -156,7 +156,7 @@ const lookupData = {
"env_vars": "SGLANG_ENABLE_JIT_DEEPGEMM=false",
"model_path": "deepseek-ai/DeepSeek-R1-0528",
"tensor_parallel_size": 8,
"cuda_graph_max_bs": 128,
"cuda_graph_max_bs_decode": 128,
"max_running_requests": 128,
"mem_fraction_static": 0.82,
"kv_cache_dtype": "fp8_e4m3",
@@ -176,7 +176,7 @@ const lookupData = {
"env_vars": "SGLANG_ENABLE_JIT_DEEPGEMM=false",
"model_path": "deepseek-ai/DeepSeek-R1-0528",
"tensor_parallel_size": 8,
"cuda_graph_max_bs": 128,
"cuda_graph_max_bs_decode": 128,
"max_running_requests": 128,
"mem_fraction_static": 0.82,
"kv_cache_dtype": "fp8_e4m3",
@@ -258,7 +258,7 @@ const lookupData = {
"tensor_parallel_size": 8,
"disable_radix_cache": true,
"max_running_requests": 256,
"cuda_graph_max_bs": 256,
"cuda_graph_max_bs_decode": 256,
"chunked_prefill_size": 32768,
"max_prefill_tokens": 32768,
"mem_fraction_static": 0.82,
@@ -278,7 +278,7 @@ const lookupData = {
"tensor_parallel_size": 8,
"disable_radix_cache": true,
"max_running_requests": 512,
"cuda_graph_max_bs": 512,
"cuda_graph_max_bs_decode": 512,
"chunked_prefill_size": 32768,
"max_prefill_tokens": 32768,
"mem_fraction_static": 0.82,
@@ -298,7 +298,7 @@ const lookupData = {
"trust_remote_code": true,
"tensor_parallel_size": 8,
"mem_fraction_static": 0.8,
"cuda_graph_max_bs": 128,
"cuda_graph_max_bs_decode": 128,
"chunked_prefill_size": 131072,
"num_continuous_decode_steps": 4,
"max_prefill_tokens": 131072,
@@ -318,7 +318,7 @@ const lookupData = {
"trust_remote_code": true,
"tensor_parallel_size": 8,
"mem_fraction_static": 0.8,
"cuda_graph_max_bs": 512,
"cuda_graph_max_bs_decode": 512,
"chunked_prefill_size": 131072,
"num_continuous_decode_steps": 4,
"max_prefill_tokens": 131072,
@@ -338,7 +338,7 @@ const lookupData = {
"trust_remote_code": true,
"tensor_parallel_size": 8,
"mem_fraction_static": 0.8,
"cuda_graph_max_bs": 128,
"cuda_graph_max_bs_decode": 128,
"chunked_prefill_size": 131072,
"num_continuous_decode_steps": 4,
"max_prefill_tokens": 131072,
@@ -358,7 +358,7 @@ const lookupData = {
"trust_remote_code": true,
"tensor_parallel_size": 8,
"mem_fraction_static": 0.8,
"cuda_graph_max_bs": 512,
"cuda_graph_max_bs_decode": 512,
"chunked_prefill_size": 131072,
"num_continuous_decode_steps": 4,
"max_prefill_tokens": 131072,
@@ -382,7 +382,7 @@ const lookupData = {
"chunked_prefill_size": 196608,
"num_continuous_decode_steps": 4,
"max_prefill_tokens": 196608,
"cuda_graph_max_bs": 128,
"cuda_graph_max_bs_decode": 128,
"attention_backend": "aiter",
"kv_cache_dtype": "fp8_e4m3"
}
@@ -402,7 +402,7 @@ const lookupData = {
"chunked_prefill_size": 196608,
"num_continuous_decode_steps": 4,
"max_prefill_tokens": 196608,
"cuda_graph_max_bs": 512,
"cuda_graph_max_bs_decode": 512,
"attention_backend": "aiter",
"kv_cache_dtype": "fp8_e4m3"
}
@@ -422,7 +422,7 @@ const lookupData = {
"chunked_prefill_size": 196608,
"num_continuous_decode_steps": 4,
"max_prefill_tokens": 196608,
"cuda_graph_max_bs": 128,
"cuda_graph_max_bs_decode": 128,
"attention_backend": "aiter",
"kv_cache_dtype": "fp8_e4m3"
}
@@ -442,7 +442,7 @@ const lookupData = {
"chunked_prefill_size": 196608,
"num_continuous_decode_steps": 4,
"max_prefill_tokens": 196608,
"cuda_graph_max_bs": 512,
"cuda_graph_max_bs_decode": 512,
"attention_backend": "aiter",
"kv_cache_dtype": "fp8_e4m3"
}
@@ -463,7 +463,7 @@ const fieldToFlag = {
tensor_parallel_size: 'tp',
data_parallel_size: 'dp',
ep_size: 'ep-size',
cuda_graph_max_bs: 'cuda-graph-max-bs',
cuda_graph_max_bs_decode: 'cuda-graph-max-bs-decode',
max_running_requests: 'max-running-requests',
mem_fraction_static: 'mem-fraction-static',
kv_cache_dtype: 'kv-cache-dtype',
docs_new/src/snippets/autoregressive/deepseek-v32-deployment.jsx
+1 -1
View File
@@ -209,7 +209,7 @@ export const DeepSeekV32Deployment = () => {
cmd += ' \\\n --trust-remote-code';
cmd += ' \\\n --dsa-prefill-backend tilelang';
cmd += ' \\\n --dsa-decode-backend tilelang';
cmd += ' \\\n --cuda-graph-max-bs 64';
cmd += ' \\\n --cuda-graph-max-bs-decode 64';
}
// Strategy configurations
docs_new/src/snippets/autoregressive/llama4-maverick-deployment.jsx
+1 -1
View File
@@ -80,7 +80,7 @@ export const Llama4MaverickDeployment = () => {
cmd += ` --speculative-eagle-topk 1 \\\n`;
cmd += ` --speculative-num-draft-tokens 4 \\\n`;
cmd += ` --mem-fraction-static 0.75 \\\n`;
cmd += ` --cuda-graph-max-bs 2`;
cmd += ` --cuda-graph-max-bs-decode 2`;
}
cmd += ` \\\n --enable-multimodal`;
docs_new/src/snippets/autoregressive/llama4-scout-deployment.jsx
+1 -1
View File
@@ -81,7 +81,7 @@ export const Llama4ScoutDeployment = () => {
cmd += ` --speculative-eagle-topk 1 \\\n`;
cmd += ` --speculative-num-draft-tokens 4 \\\n`;
cmd += ` --mem-fraction-static 0.75 \\\n`;
cmd += ` --cuda-graph-max-bs 2`;
cmd += ` --cuda-graph-max-bs-decode 2`;
}
cmd += ` \\\n --enable-multimodal`;
docs_new/src/snippets/autoregressive/mimo-v25-deployment.jsx
+1 -1
View File
@@ -344,7 +344,7 @@ export const MiMoV25Deployment = () => {
flags.push(" --mem-fraction-static 0.7");
flags.push(" --max-running-requests 128");
flags.push(" --chunked-prefill-size 32768");
flags.push(" --cuda-graph-max-bs 64");
flags.push(" --cuda-graph-max-bs-decode 64");
flags.push(" --page-size 64");
flags.push(" --swa-full-tokens-ratio 0.3");
flags.push(` --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 64}'`);
docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx
+16 -16
View File
@@ -438,7 +438,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.92",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -458,7 +458,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.835",
"--cuda-graph-max-bs 544",
"--cuda-graph-max-bs-decode 544",
"--swa-full-tokens-ratio 0.075",
"--chunked-prefill-size 65536",
"--tokenizer-worker-num 8",
@@ -565,7 +565,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.92",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -585,7 +585,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.835",
"--cuda-graph-max-bs 544",
"--cuda-graph-max-bs-decode 544",
"--swa-full-tokens-ratio 0.075",
"--chunked-prefill-size 65536",
"--tokenizer-worker-num 8",
@@ -789,7 +789,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.78",
"--cuda-graph-max-bs 64",
"--cuda-graph-max-bs-decode 64",
"--max-running-requests 128",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -811,7 +811,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.78",
"--cuda-graph-max-bs 64",
"--cuda-graph-max-bs-decode 64",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -959,7 +959,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 256",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
@@ -980,7 +980,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1068,7 +1068,7 @@ sgl-eval run aime25 \\
"--speculative-num-steps 1",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 128",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
@@ -1089,7 +1089,7 @@ sgl-eval run aime25 \\
"--dp 4",
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 256",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
@@ -1110,7 +1110,7 @@ sgl-eval run aime25 \\
"--dp 16",
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--cuda-graph-max-bs 8",
"--cuda-graph-max-bs-decode 8",
"--max-running-requests 32",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 3",
@@ -1140,7 +1140,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.88",
"--cuda-graph-max-bs 8",
"--cuda-graph-max-bs-decode 8",
"--max-running-requests 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1161,7 +1161,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.88",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1330,7 +1330,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs 8",
"--cuda-graph-max-bs-decode 8",
"--max-running-requests 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1350,7 +1350,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs 8",
"--cuda-graph-max-bs-decode 8",
"--max-running-requests 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1385,7 +1385,7 @@ sgl-eval run aime25 \\
"--tp 4",
"--moe-runner-backend marlin",
"--mem-fraction-static 0.70",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
docs_new/src/snippets/configs/zai-org/glm-5.2.jsx
+24 -24
View File
@@ -185,7 +185,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
"--mem-fraction-static 0.8",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -205,7 +205,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
// Large chunked-prefill is the dominant balanced lever (prefill-bound at this
// concurrency); max-running tracks KV capacity (~60-80 for 8K+1K reqs on 8xH200).
"--chunked-prefill-size 32768",
@@ -225,7 +225,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -247,7 +247,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
"--mem-fraction-static 0.8",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -267,7 +267,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
// Large chunked-prefill is the dominant balanced lever (prefill-bound at this
// concurrency); max-running tracks KV capacity (~89 for 8K+1K reqs on 8xB200).
"--chunked-prefill-size 32768",
@@ -287,7 +287,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -312,7 +312,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -332,7 +332,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
// Same prefill lever as H200/B200 balanced; max-running tracks the TP4 KV capacity.
"--chunked-prefill-size 32768",
"--max-running-requests 80",
@@ -351,7 +351,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -375,7 +375,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
"--mem-fraction-static 0.8",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -395,7 +395,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--chunked-prefill-size 32768",
"--max-running-requests 80",
"--host {{HOST_IP}}",
@@ -413,7 +413,7 @@ sgl-eval run aime25 \\
"--enable-dp-attention",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -438,7 +438,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -455,7 +455,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--chunked-prefill-size 32768",
"--max-running-requests 80",
"--host {{HOST_IP}}",
@@ -470,7 +470,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -495,7 +495,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -512,7 +512,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--chunked-prefill-size 32768",
"--max-running-requests 80",
"--host {{HOST_IP}}",
@@ -527,7 +527,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -545,7 +545,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -562,7 +562,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--chunked-prefill-size 32768",
"--max-running-requests 80",
"--host {{HOST_IP}}",
@@ -577,7 +577,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -595,7 +595,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -612,7 +612,7 @@ sgl-eval run aime25 \\
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 2",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--chunked-prefill-size 32768",
"--max-running-requests 80",
"--host {{HOST_IP}}",
@@ -627,7 +627,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--host {{HOST_IP}}",
"--port {{PORT}}",
examples/runtime/engine/offline_batch_inference_eagle.py
+1 -1
View File
@@ -21,7 +21,7 @@ def main():
speculative_num_steps=3,
speculative_eagle_topk=4,
speculative_num_draft_tokens=16,
cuda_graph_max_bs=8,
cuda_graph_max_bs_decode=8,
)
outputs = llm.generate(prompts, sampling_params)
python/sglang/test/ascend/vlm_utils.py
+1 -1
View File
@@ -17,7 +17,7 @@ class TestVLMModels(CustomTestCase):
mmmu_accuracy = 0.00
other_args = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--enable-multimodal",
"--mem-fraction-static",
python/sglang/test/doc_patch.py
+1 -1
View File
@@ -52,7 +52,7 @@ def launch_server_cmd(command: str, host: str = "0.0.0.0", port: int = None):
f"--max-running-requests {DEFAULT_MAX_RUNNING_REQUESTS} "
f"--max-total-tokens {DEFAULT_MAX_TOTAL_TOKENS} "
f"--disable-cuda-graph "
f"--cuda-graph-max-bs 4"
f"--cuda-graph-max-bs-decode 4"
)
full_command = f"{command} --port {port} {extra_flags}"
python/sglang/test/gpt_oss_common.py
+1 -1
View File
@@ -40,7 +40,7 @@ class BaseTestGptOss(CustomTestCase):
}[(model_variant, quantization)]
if model_variant == "20b":
other_args += ["--cuda-graph-max-bs", "600"]
other_args += ["--cuda-graph-max-bs-decode", "600"]
# Respect SGLANG_USE_AITER if already set, otherwise default to "0" for HIP
if _is_hip and "SGLANG_USE_AITER" not in os.environ:
os.environ["SGLANG_USE_AITER"] = "0"
python/sglang/test/kits/mmmu_vlm_kit.py
+1 -1
View File
@@ -367,7 +367,7 @@ class MMMUMultiModelTestBase(CustomTestCase):
api_key=self.api_key,
other_args=[
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"64",
"--enable-multimodal",
"--mem-fraction-static",
python/sglang/test/kv_canary/consts.py
+1 -1
View File
@@ -28,7 +28,7 @@ DSV4_POOL_SERVER_ARGS: Final[tuple[str, ...]] = (
"--enable-dp-attention",
"--moe-a2a-backend",
"deepep",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
python/sglang/test/server_fixtures/hybrid_attn_backend_fixture.py
+1 -1
View File
@@ -28,7 +28,7 @@ GSM_DATASET_PATH = None
# Default server arguments shared across all hybrid-attn-backend tests
DEFAULT_HYBRID_ATTN_SERVER_ARGS = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--prefill-attention-backend",
"fa3",
python/sglang/test/server_fixtures/mmmu_fixture.py
+1 -1
View File
@@ -44,7 +44,7 @@ class MMMUServerBase(CustomTestCase):
# Build server args with MMMU-specific settings
server_args = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"64",
"--enable-multimodal",
"--mem-fraction-static",
python/sglang/test/server_fixtures/ngram_fixture.py
+1 -1
View File
@@ -26,7 +26,7 @@ from sglang.test.test_utils import (
DEFAULT_NGRAM_SERVER_ARGS = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--speculative-algorithm",
"NGRAM",
python/sglang/test/server_fixtures/spec_eagle_fixture.py
+3 -3
View File
@@ -64,7 +64,7 @@ class SpecEagleServerBase(CustomTestCase):
# bf16 rather than fp16: fp16 activations can overflow (-> Inf -> NaN) on
# degenerate draft branches in verify and trip the CI NaN asserts.
dtype = "bfloat16"
cuda_graph_max_bs = None
cuda_graph_max_bs_decode = None
trust_remote_code = True
# Launch with --enable-return-hidden-states so SpecHiddenStatesKit can probe
# per-request hidden states; per-request gated, so other requests don't pay.
@@ -109,8 +109,8 @@ class SpecEagleServerBase(CustomTestCase):
args.append("--trust-remote-code")
if cls.enable_return_hidden_states:
args.append("--enable-return-hidden-states")
if cls.cuda_graph_max_bs is not None:
args += ["--cuda-graph-max-bs", str(cls.cuda_graph_max_bs)]
if cls.cuda_graph_max_bs_decode is not None:
args += ["--cuda-graph-max-bs-decode", str(cls.cuda_graph_max_bs_decode)]
args += [str(a) for a in cls.extra_args]
return args
python/sglang/test/server_fixtures/standalone_fixture.py
+1 -1
View File
@@ -49,7 +49,7 @@ class StandaloneServerBase:
assert cls.attention_backend, f"{cls.__name__} must set `attention_backend`"
args = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--speculative-algorithm",
"STANDALONE",
python/sglang/test/test_deterministic_utils.py
+1 -1
View File
@@ -12,7 +12,7 @@ from sglang.test.test_utils import (
DEFAULT_MODEL = "Qwen/Qwen3-8B"
COMMON_SERVER_ARGS = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--enable-deterministic-inference",
]
test/manual/attention/test_fa3.py
+5 -5
View File
@@ -45,7 +45,7 @@ if OFFLINE_MODE:
# Default server arguments shared across all tests
DEFAULT_SERVER_ARGS = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--attention-backend",
"fa3",
@@ -144,7 +144,7 @@ class TestFlashAttention3SpeculativeDecode(BaseFlashAttentionTest):
args = DEFAULT_SERVER_ARGS
args.extend(
[
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"4",
"--speculative-algorithm",
"EAGLE3",
@@ -178,7 +178,7 @@ class TestFlashAttention3SpeculativeDecodeTopk(BaseFlashAttentionTest):
args = DEFAULT_SERVER_ARGS
args.extend(
[
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"4",
"--speculative-algorithm",
"EAGLE3",
@@ -210,7 +210,7 @@ class TestFlashAttention3MLASpeculativeDecode(BaseFlashAttentionTest):
args = DEFAULT_SERVER_ARGS
args.extend(
[
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"4",
"--speculative-algorithm",
"EAGLE",
@@ -242,7 +242,7 @@ class TestFlashAttention3MLASpeculativeDecodeTopk(BaseFlashAttentionTest):
args = DEFAULT_SERVER_ARGS
args.extend(
[
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"4",
"--speculative-algorithm",
"EAGLE",
test/manual/attention/test_local_attn.py
+1 -1
View File
@@ -25,7 +25,7 @@ class TestFlashAttention3LocalAttn(CustomTestCase):
@classmethod
def get_server_args(cls):
return [
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"2",
"--attention-backend",
"fa3",
test/manual/dsv4/test_b200_pro.py
+3 -3
View File
@@ -60,7 +60,7 @@ class TestB200ProBalanced(DSV4ProAime25TestBase):
"2",
"--mem-fraction-static",
"0.82",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"64",
"--max-running-requests",
"128",
@@ -83,7 +83,7 @@ class TestB200ProMaxThroughput(DSV4ProAime25TestBase):
"deepep",
"--mem-fraction-static",
"0.82",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"64",
"--max-running-requests",
"256",
@@ -108,7 +108,7 @@ class TestB200ProCP(DSV4ProAime25TestBase):
"16384",
"--mem-fraction-static",
"0.78",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"256",
"--max-running-requests",
"256",
test/manual/dsv4/test_b300_pro.py
+3 -3
View File
@@ -62,7 +62,7 @@ class TestB300ProBalanced(DSV4ProAime25TestBase):
"2",
"--mem-fraction-static",
"0.82",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"64",
"--max-running-requests",
"128",
@@ -85,7 +85,7 @@ class TestB300ProMaxThroughput(DSV4ProAime25TestBase):
"deepep",
"--mem-fraction-static",
"0.82",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"64",
"--max-running-requests",
"256",
@@ -110,7 +110,7 @@ class TestB300ProCP(DSV4ProAime25TestBase):
"16384",
"--mem-fraction-static",
"0.78",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"256",
"--max-running-requests",
"256",
test/manual/dsv4/test_dsv4_flash_mtp_dp4.py
+2 -2
View File
@@ -37,7 +37,7 @@ DSV4_FLASH_ENV = {
"SGLANG_DSV4_FP4_EXPERTS": "0",
# MTP runs ~num_draft_tokens forward passes per step, so the deepep
# dispatch input size scales by that factor. Default 256 (used by the
# plain server) overflows once cuda-graph-max-bs * num_draft_tokens
# plain server) overflows once cuda-graph-max-bs-decode * num_draft_tokens
# > 256. 1024 covers bs=128 * 4 draft tokens with headroom.
"SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK": "1024",
}
@@ -66,7 +66,7 @@ class DSV4FlashMTPServerBase(CustomTestCase):
"--enable-dp-attention",
"--moe-a2a-backend",
"deepep",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
test/manual/dsv4/test_dsv4_flash_sanity_dp4.py
+3 -3
View File
@@ -65,7 +65,7 @@ class TestDSV4FlashTP4DP4(
"--enable-dp-attention",
"--moe-a2a-backend",
"deepep",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
@@ -101,7 +101,7 @@ class TestDSV4FlashTP4EP(
# No --enable-dp-attention by design: covers TP-attn path.
"--moe-a2a-backend",
"deepep",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"64",
@@ -139,7 +139,7 @@ class TestDSV4FlashTP4DP4ChunkedPrefillLarge(
"deepep",
"--chunked-prefill-size",
"16384",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
test/manual/dsv4/test_dsv4_pd_disagg_nixl.py
+2 -2
View File
@@ -61,7 +61,7 @@ class TestDSV4FlashPDDisaggNIXL(PDDisaggregationServerBase):
"deepep",
"--deepep-config",
DEEPEP_CONFIG,
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
@@ -103,7 +103,7 @@ class TestDSV4FlashPDDisaggNIXL(PDDisaggregationServerBase):
"deepep",
"--deepep-config",
DEEPEP_CONFIG,
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
test/manual/dsv4/test_dsv4_swa_radix_retract.py
+1 -1
View File
@@ -66,7 +66,7 @@ class TestDSV4FlashSWARadixRetract(CustomTestCase):
"--enable-dp-attention",
"--moe-a2a-backend",
"deepep",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
test/manual/dsv4/test_gb300_pro.py
+3 -3
View File
@@ -62,7 +62,7 @@ class TestGB300ProBalanced(DSV4ProAime25TestBase):
"2",
"--mem-fraction-static",
"0.9",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
@@ -85,7 +85,7 @@ class TestGB300ProMaxThroughput(DSV4ProAime25TestBase):
"deepep",
"--mem-fraction-static",
"0.9",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
@@ -110,7 +110,7 @@ class TestGB300ProCP(DSV4ProAime25TestBase):
"16384",
"--mem-fraction-static",
"0.88",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"256",
"--max-running-requests",
"256",
test/manual/dsv4/test_h200_fp8_flash.py
+2 -2
View File
@@ -53,7 +53,7 @@ class TestH200Fp8FlashBalanced(DSV4FlashAime25TestBase):
"1",
"--speculative-num-draft-tokens",
"2",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"128",
@@ -77,7 +77,7 @@ class TestH200Fp8FlashMaxThroughput(DSV4FlashAime25TestBase):
"--enable-dp-attention",
"--moe-a2a-backend",
"deepep",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
test/manual/dsv4/test_h200_fp8_pro.py
+3 -3
View File
@@ -44,7 +44,7 @@ class TestH200Fp8ProLowLatency(DSV4ProAime25TestBase):
*multinode_args(2),
"--moe-a2a-backend",
"deepep",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--max-running-requests",
"32",
@@ -88,7 +88,7 @@ class TestH200Fp8ProBalanced(DSV4ProAime25TestBase):
"2",
"--mem-fraction-static",
"0.88",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--max-running-requests",
"32",
@@ -114,7 +114,7 @@ class TestH200Fp8ProMaxThroughput(DSV4ProAime25TestBase):
"deepep",
"--mem-fraction-static",
"0.88",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"256",
test/manual/ep/test_eplb_mori.py
+1 -1
View File
@@ -69,7 +69,7 @@ common_args = [
"12288",
"--attention-backend",
"aiter",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
]
test/manual/ep/test_moe_deepep_eval_accuracy_large.py
+1 -1
View File
@@ -32,7 +32,7 @@ class TestMoEDeepEPEvalAccuracyLarge(CustomTestCase):
"8",
"--moe-a2a-backend",
"deepep",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
],
)
test/manual/ep/test_mooncake_expert_backup.py
+1 -1
View File
@@ -58,7 +58,7 @@ class TestBackup(CustomTestCase):
"50",
"--chunked-prefill-size",
"512",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-running-requests",
"512",
test/manual/mla/test_flashmla_archived.py
+1 -1
View File
@@ -35,7 +35,7 @@ class TestFlashMLAAttnBackend(unittest.TestCase):
if torch.cuda.is_available() and torch.version.cuda:
other_args.extend(
[
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"2",
"--attention-backend",
"flashmla",
test/manual/mla/test_mla_deepseek_v3.py
+6 -4
View File
@@ -22,7 +22,9 @@ class TestMLADeepseekV3(CustomTestCase):
cls.base_url = DEFAULT_URL_FOR_TEST
other_args = ["--trust-remote-code", "--chunked-prefill-size", "256"]
if is_cuda():
other_args.extend(["--enable-torch-compile", "--cuda-graph-max-bs", "2"])
other_args.extend(
["--enable-torch-compile", "--cuda-graph-max-bs-decode", "2"]
)
cls.process = popen_launch_server(
cls.model,
cls.base_url,
@@ -59,7 +61,7 @@ class TestMLADeepseekV3DisableFusedFunc(CustomTestCase):
cls.base_url = DEFAULT_URL_FOR_TEST
other_args = ["--trust-remote-code", "--chunked-prefill-size", "256"]
if is_cuda():
other_args.extend(["--cuda-graph-max-bs", "2"])
other_args.extend(["--cuda-graph-max-bs-decode", "2"])
cls.process = popen_launch_server(
cls.model,
cls.base_url,
@@ -107,7 +109,7 @@ class TestMLADeepseekV3Fa3Fp8Kvcache(CustomTestCase):
"fa3",
"--mem-fraction-static",
"0.8",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"2",
]
)
@@ -145,7 +147,7 @@ class TestDeepseekV3MTP(CustomTestCase):
cls.base_url = DEFAULT_URL_FOR_TEST
other_args = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"2",
"--disable-radix",
"--enable-torch-compile",
test/manual/mla/test_mla_flashinfer_archived.py
+1 -1
View File
@@ -31,7 +31,7 @@ class TestFlashinferMLA(CustomTestCase):
other_args.extend(
[
"--enable-torch-compile",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"4",
"--attention-backend",
"flashinfer",
test/manual/mla/test_mla_int8_deepseek_v3_archived.py
+2 -2
View File
@@ -31,7 +31,7 @@ class TestMLADeepseekV3ChannelInt8(CustomTestCase):
if torch.cuda.is_available() and torch.version.cuda:
other_args.extend(
[
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"16",
"--enable-torch-compile",
"--torch-compile-max-bs",
@@ -75,7 +75,7 @@ class TestMLADeepseekV3BlockInt8(CustomTestCase):
if torch.cuda.is_available() and torch.version.cuda:
other_args.extend(
[
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"16",
"--enable-torch-compile",
"--torch-compile-max-bs",
test/manual/perf/test_bench_one_batch_1gpu.py
+2 -2
View File
@@ -20,7 +20,7 @@ class TestBenchOneBatch1GPU(CustomTestCase):
def test_bs1_small(self):
_, output_throughput, _ = run_bench_one_batch(
DEFAULT_SMALL_MODEL_NAME_FOR_TEST, ["--cuda-graph-max-bs", "2"]
DEFAULT_SMALL_MODEL_NAME_FOR_TEST, ["--cuda-graph-max-bs-decode", "2"]
)
self.assertGreater(output_throughput, 50)
@@ -42,7 +42,7 @@ class TestBenchOneBatch1GPU(CustomTestCase):
"1024",
"--model-path",
DEFAULT_MODEL_NAME_FOR_TEST,
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"2",
]
test/manual/test_kv_events.py
+3 -3
View File
@@ -46,7 +46,7 @@ class TestKvEvents(CustomTestCase):
'{"publisher": "zmq", "topic": "kv-events"}',
"--max-total-tokens",
32,
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
2,
"--enable-dp-attention",
"--dp-size",
@@ -190,7 +190,7 @@ class TestKvEvents(CustomTestCase):
'{"publisher": "zmq", "topic": "kv-events"}',
"--max-total-tokens",
64,
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
4,
"--enable-dp-attention",
"--dp-size",
@@ -327,7 +327,7 @@ class TestKvEvents(CustomTestCase):
"--max-running-requests",
4,
"--disable-cuda-graph",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
4,
"--model-loader-extra-config",
'{"enable_multithread_load": true, "num_threads": 64}',
test/manual/test_mla_tp.py
+7 -1
View File
@@ -21,7 +21,13 @@ class TestDeepseekTP2(CustomTestCase):
other_args = ["--trust-remote-code"]
if torch.cuda.is_available() and torch.version.cuda:
other_args.extend(
["--tp", "2", "--enable-torch-compile", "--cuda-graph-max-bs", "2"]
[
"--tp",
"2",
"--enable-torch-compile",
"--cuda-graph-max-bs-decode",
"2",
]
)
cls.process = popen_launch_server(
cls.model,
test/manual/test_qwen3_235b.py
+1 -1
View File
@@ -84,7 +84,7 @@ class TestQwen3235BFP8(unittest.TestCase):
]
MTP_ARGS = [
"--cuda-graph-max-bs=32",
"--cuda-graph-max-bs-decode=32",
"--max-running-requests=32",
]
variants = [
test/manual/test_vertex_endpoint.py
+1 -1
View File
@@ -26,7 +26,7 @@ class TestVertexEndpoint(CustomTestCase):
cls.model,
cls.base_url,
timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
other_args=["--cuda-graph-max-bs", 2],
other_args=["--cuda-graph-max-bs-decode", 2],
)
@classmethod
test/manual/vlm/test_anthropic_vision.py
+1 -1
View File
@@ -44,7 +44,7 @@ class TestAnthropicVision(CustomTestCase):
other_args=[
"--trust-remote-code",
"--enable-multimodal",
"--cuda-graph-max-bs=4",
"--cuda-graph-max-bs-decode=4",
],
)
cls.messages_url = cls.base_url + "/v1/messages"
test/registered/8-gpu-models/test_gpt_oss_120b.py
+2 -2
View File
@@ -26,14 +26,14 @@ class TestGptOss120B(unittest.TestCase):
base_args = [
"--tp=8",
"--trust-remote-code",
"--cuda-graph-max-bs=200",
"--cuda-graph-max-bs-decode=200",
"--mem-fraction-static=0.93",
]
# Lower batch size for EAGLE3 variants to avoid OOM
base_args_eagle3 = [
"--tp=8",
"--trust-remote-code",
"--cuda-graph-max-bs=100",
"--cuda-graph-max-bs-decode=100",
"--mem-fraction-static=0.85",
]
parser_args = [
test/registered/amd/accuracy/mi30x/test_deepseek_r1_eval_amd.py
+1 -1
View File
@@ -138,7 +138,7 @@ DEEPSEEK_R1_MODELS = [
"131072",
"--mem-fraction-static",
"0.70",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--enable-torch-compile",
"--disable-cuda-graph",
test/registered/amd/accuracy/mi30x/test_deepseek_v32_tc_eval_amd.py
+1 -1
View File
@@ -56,7 +56,7 @@ class TestDeepseekV32TC(CustomTestCase):
"131072",
"--mem-fraction-static",
"0.70",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--enable-torch-compile",
"--disable-cuda-graph",
test/registered/amd/test_moriep_small.py
+1 -1
View File
@@ -72,7 +72,7 @@ common_args = [
"32768",
"--attention-backend",
"aiter",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
]
test/registered/ascend/basic_function/runtime_opts/test_npu_tp4_bf16.py
+1 -1
View File
@@ -39,7 +39,7 @@ class TestAscendTp4Bf16(CustomTestCase):
"--attention-backend",
"ascend",
"--disable-radix-cache",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
32,
"--tp-size",
4,
test/registered/ascend/llm_models/test_npu_llama_2_7b_communications_compression.py
+1 -1
View File
@@ -25,7 +25,7 @@ class TestLlama(GSM8KAscendMixin, CustomTestCase):
32,
"--attention-backend",
"ascend",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
32,
"--tp-size",
2,
test/registered/ascend/llm_models/test_npu_qwen3_30b.py
+1 -1
View File
@@ -28,7 +28,7 @@ class TestQwen330B(GSM8KAscendMixin, CustomTestCase):
"--attention-backend",
"ascend",
"--disable-cuda-graph",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
32,
"--tp-size",
2,
test/registered/ascend/llm_models/test_npu_qwen3_30b_attn_cp.py
+1 -1
View File
@@ -37,7 +37,7 @@ class TestQwen330BAttnCP(GSM8KAscendMixin, CustomTestCase):
"2",
"--attn-cp-size",
"2",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--enable-prefill-context-parallel",
]
test/registered/ascend/llm_models/test_npu_qwen3_30b_w4a4.py
+1 -1
View File
@@ -26,7 +26,7 @@ class TestQwen317BGPTQInt8(GSM8KAscendMixin, CustomTestCase):
"--attention-backend",
"ascend",
"--disable-cuda-graph",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
32,
"--tp-size",
2,
test/registered/ascend/llm_models/test_npu_qwen3_8b_communications_quantization.py
+1 -1
View File
@@ -25,7 +25,7 @@ class TestQwen38BCommQuantization(GSM8KAscendMixin, CustomTestCase):
32,
"--attention-backend",
"ascend",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
32,
"--tp-size",
2,
test/registered/ascend/vlm_models/test_ascend_glm_4_5v.py
+1 -1
View File
@@ -11,7 +11,7 @@ class TestGLM4Models(TestVLMModels):
mmmu_accuracy = 0.2
other_args = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--enable-multimodal",
"--mem-fraction-static",
test/registered/ascend/vlm_models/test_npu_qwen2_5_vl_72b_instruct.py
+1 -1
View File
@@ -18,7 +18,7 @@ class TestQwen25VL72B(TestVLMModels):
mmmu_accuracy = 0.2
other_args = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--enable-multimodal",
"--mem-fraction-static",
test/registered/ascend/vlm_models/test_npu_qwen3_vl_235b_a22b_instruct.py
+1 -1
View File
@@ -20,7 +20,7 @@ class TestQwen3VL235BA22B(TestVLMModels):
mmmu_accuracy = 0.2
other_args = [
"--trust-remote-code",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--enable-multimodal",
"--mem-fraction-static",
test/registered/attention/test_triton_attention_backend.py
+1 -1
View File
@@ -32,7 +32,7 @@ class TestTritonAttnBackend(CustomTestCase):
"--attention-backend",
"triton",
"--enable-torch-compile",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
4,
],
)
test/registered/backends/test_deepseek_r1_fp8_trtllm_backend.py
+1 -1
View File
@@ -31,7 +31,7 @@ class TestDeepseekR1Fp8Flashinfer(CustomTestCase):
"8192",
"--mem-fraction-static",
"0.9",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--max-prefill-tokens",
"8192",
test/registered/backends/test_flashinfer_trtllm_gen_attn_backend.py
+1 -1
View File
@@ -28,7 +28,7 @@ class TestFlashinferTrtllmGenAttnBackend(CustomTestCase):
other_args=[
"--attention-backend",
"trtllm_mha",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"512",
"--tp-size",
"4",
test/registered/backends/test_flashinfer_trtllm_gen_moe_backend.py
+2 -2
View File
@@ -78,7 +78,7 @@ class FlashinferTrtllmGenMoeBackendBF16Base:
"triton",
"--moe-runner-backend",
cls.backend,
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"512",
"--tp-size",
"4",
@@ -261,7 +261,7 @@ class FlashinferTrtllmGenMoeBackendNvFp4OnlineBase:
"triton",
"--moe-runner-backend",
cls.backend,
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"128",
"--tp-size",
"4",
test/registered/backends/test_torch_compile.py
+1 -1
View File
@@ -32,7 +32,7 @@ class TestTorchCompile(CustomTestCase, MMLUMixin):
cls.model,
cls.base_url,
timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
other_args=["--enable-torch-compile", "--cuda-graph-max-bs", "4"],
other_args=["--enable-torch-compile", "--cuda-graph-max-bs-decode", "4"],
)
@classmethod
test/registered/core/test_basic_sanity.py
+1 -1
View File
@@ -48,7 +48,7 @@ class TestBasicSanity(
cls.base_url,
timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
other_args=[
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"4",
"--mem-fraction-static",
"0.7",
test/registered/core/test_basic_sanity_eagle3.py
+1 -1
View File
@@ -70,7 +70,7 @@ class TestBasicSanityEagle3(
"1",
"--speculative-num-draft-tokens",
"2",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"4",
"--mem-fraction-static",
"0.7",
test/registered/core/test_srt_endpoint.py
+1 -1
View File
@@ -48,7 +48,7 @@ class TestSRTEndpoint(CustomTestCase):
"--enable-custom-logit-processor",
"--mem-fraction-static",
"0.7",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
),
)
test/registered/cp/test_deepseek_v32_cp_single_node.py
+2 -2
View File
@@ -44,7 +44,7 @@ class TestDeepseekV32CPInSeqSplit(CustomTestCase):
"4",
"--mem-frac",
"0.85",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--max-running-requests",
"32",
@@ -110,7 +110,7 @@ class TestDeepseekV32CPRoundRobinSplit(CustomTestCase):
"4",
"--mem-frac",
"0.85",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--max-running-requests",
"32",
test/registered/cp/test_deepseek_v3_cp_single_node.py
+1 -1
View File
@@ -42,7 +42,7 @@ class TestDeepseekV3CPInSeqSplit(CustomTestCase):
"fa3",
"--mem-frac",
"0.7",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--max-running-requests",
"32",
test/registered/cp/test_gqa_prefill_cp_legacy.py
+2 -2
View File
@@ -37,7 +37,7 @@ class TestGQACP2TP2EP2(CustomTestCase):
"--attn-cp-size",
"2",
"--enable-prefill-context-parallel",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--max-running-requests",
"32",
@@ -94,7 +94,7 @@ class TestGQACPTP2CP2EP4(CustomTestCase):
"--attn-cp-size",
"2",
"--enable-prefill-context-parallel",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--max-running-requests",
"32",
test/registered/cp/test_gqa_preill_cp.py
+3 -3
View File
@@ -39,7 +39,7 @@ class TestGQACP2TP2EP2(CustomTestCase):
"--enable-prefill-cp",
"--cp-strategy",
"zigzag",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--max-running-requests",
"32",
@@ -98,7 +98,7 @@ class TestGQACPTP2CP2EP4(CustomTestCase):
"--enable-prefill-cp",
"--cp-strategy",
"zigzag",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--max-running-requests",
"32",
@@ -159,7 +159,7 @@ class TestGQACPCP4EP4(CustomTestCase):
"deepep",
"--attention-backend",
"fa3",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"32",
"--max-running-requests",
"32",
test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_extra.py
+1 -1
View File
@@ -69,7 +69,7 @@ class TestPCGWithNGRAM(PCGSpecBase, unittest.TestCase):
"NGRAM",
"--speculative-num-draft-tokens",
"16",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"8",
"--mem-fraction-static",
"0.8",
test/registered/dcp/test_dsv31_dcp8_gsm8k.py
+1 -1
View File
@@ -73,7 +73,7 @@ _COMMON_SERVER_ARGS = [
"16384",
"--max-running-requests",
"256",
"--cuda-graph-max-bs",
"--cuda-graph-max-bs-decode",
"256",
"--attention-backend",
"flashinfer",

Some files were not shown because too many files have changed in this diff Show More