minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[Config] Retire get_global_server_args, and clear the deprecated flags that have a replacement (#38375)

Browse Source
This commit is contained in:
Cheng Wan
2026-09-08 16:42:12 -07:00
committed by GitHub
parent 5177a3ec08
commit db272201a2
213 changed files with 625 additions and 892 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx
+1 -1
View File
@@ -392,7 +392,7 @@ python3 -m sglang.launch_server \
The default configuration is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Find the best values for your workload with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`.
<Note>
For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs` to include your target batch sizes (default captured sizes for speculative decoding: 48).
For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs-decode` to include your target batch sizes (default captured sizes for speculative decoding: 48).
</Note>
<Tip>
docs/cookbook/autoregressive/Meituan/LongCat-2.0.mdx
+1 -1
View File
@@ -102,7 +102,7 @@ import { Playground } from "/src/snippets/_playground.jsx";
- **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint.
- **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you.
- **LongCat sparse attention.** Keep `--nsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path.
- **LongCat sparse attention.** Keep `--dsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path.
- **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster.
- **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time.
- **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout.
docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx
+1 -1
View File
@@ -260,7 +260,7 @@ checkpoint's calibration scales automatically.
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
The Ascend comparison in #35629 used a 910C with BF16 target weights,
`--tp-size 2 --attention-backend ascend --mamba-ssm-dtype bfloat16
--mamba-scheduler-strategy extra_buffer`, and disabled RadixCache for both
--mamba-radix-cache-strategy extra_buffer`, and disabled RadixCache for both
baseline and DFlash2 to exclude cache warm-up and prefix reuse. The DFlash2
run added the three flags shown above.
Accuracy used zero-shot GSM8K with greedy sampling, `max_new_tokens=2048`,
docs/docs/advanced_features/cuda_graph_for_multi_modal_encoder.mdx
+7 -4
View File
@@ -61,14 +61,17 @@ SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
python3 -m sglang.launch_server \
--model Qwen/Qwen3-VL-8B-Instruct
```
Or you can run CUDA Graph for ViT together with Piecewise CUDA Graph feature by both setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` and setting `--enable-piecewise-cuda-graph`, for example:
To run the ViT graph together with Piecewise CUDA Graph over the decoder's
prefill, ask for the prefill backend explicitly. Qwen3-VL is on neither
multimodal allowlist, so the default backend resolves prefill to `disabled` for
it; naming a backend locks the choice and skips that rule:
```shell Command
SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
python3 -m sglang.launch_server \
--model Qwen/Qwen3-VL-8B-Instruct \
--piecewise-cuda-graph-max-tokens 4096 \
--enable-piecewise-cuda-graph \
--piecewise-cuda-graph-compiler eager
--cuda-graph-backend-prefill tc_piecewise \
--cuda-graph-max-bs-prefill 4096 \
--cuda-graph-tc-compiler eager
```
## Known supported models
docs/docs/advanced_features/pd_disaggregation.mdx
+3 -4
View File
@@ -568,7 +568,7 @@ python3 -m sglang.launch_server \
--disaggregation-bootstrap-port 8996 \
--base-gpu-id 8 \
--disable-radix-cache \
--cuda-graph-bs 1 2 4 8 10 12 14 16 \
--cuda-graph-bs-decode 1 2 4 8 10 12 14 16 \
--speculative-draft-model-quantization unquant \
--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
@@ -681,9 +681,8 @@ do
--port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \
--mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
--prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
--load-balance-method round_robin
NODE_RANK=$i
break
docs/docs/advanced_features/piecewise_cuda_graph.mdx
+14 -24
View File
@@ -10,7 +10,7 @@ Standard CUDA graphs capture the entire model forward pass as a single graph. Th
Piecewise CUDA Graph (PCG) solves this by splitting the model's computation graph into pieces (roughly one per layer) at "split points" (e.g., MoE dispatch ops). Each piece is captured as a separate CUDA graph for a set of pre-defined token lengths. At runtime, the input is padded to the nearest captured size, and each piece is replayed. This eliminates kernel launch overhead for prefill/extend while still supporting dynamic shapes.
Recently we **enabled PCG by default**, which means that the old `--enable-piecewise-cuda-graph` flag is deprecated. Use `--disable-piecewise-cuda-graph` to turn it off.
PCG is **enabled by default**. Pass `--cuda-graph-backend-prefill=disabled` to turn it off.
## Usage
@@ -26,7 +26,7 @@ python3 -m sglang.launch_server \
```bash
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disable-piecewise-cuda-graph
--cuda-graph-backend-prefill=disabled
```
### Custom capture sizes
@@ -34,7 +34,7 @@ python3 -m sglang.launch_server \
```bash
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--piecewise-cuda-graph-max-tokens 2048
--cuda-graph-max-bs-prefill 2048
```
### Server Args
@@ -54,41 +54,31 @@ python3 -m sglang.launch_server \
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--disable-piecewise-cuda-graph</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Disable PCG for extend/prefill.</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--cuda-graph-backend-prefill</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend for the prefill phase. Choices: <code>full</code>, <code>breakable</code>, <code>tc_piecewise</code>, <code>disabled</code>. Pass <code>disabled</code> to turn PCG off for extend/prefill, or <code>tc_piecewise</code> to force it on, skipping all auto-disable conditions (testing only).</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--enforce-piecewise-cuda-graph</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Force-enable PCG, skipping all auto-disable conditions. For testing only.</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-max-tokens</code></td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--cuda-graph-max-bs-prefill</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Maximum token count to capture. Defaults to <code>chunked_prefill_size</code> (non-MLA) or <code>2048</code> (MLA).</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-tokens</code></td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--cuda-graph-bs-prefill</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Explicit list of token lengths to capture. Auto-generated if not set.</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-compiler</code></td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--cuda-graph-tc-compiler</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>"eager"</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Compiler backend for the captured subgraphs. Choices: <code>eager</code>, <code>inductor</code>.</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><del><code>--enable-piecewise-cuda-graph</code></del></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Deprecated.</strong> PCG is now enabled by default. Use <code>--enforce-piecewise-cuda-graph</code> to skip auto-disable conditions.</td>
</tr>
</tbody>
</table>
## Bug Report
PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--disable-piecewise-cuda-graph` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature.
PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--cuda-graph-backend-prefill=disabled` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature.
### For Users
@@ -96,11 +86,11 @@ If you see an error message like the following during server startup, it is a PC
```
Piecewise CUDA Graph is enabled by default as an experimental feature.
To work around this error, add --disable-piecewise-cuda-graph to your launch command.
To work around this error, add --cuda-graph-backend-prefill=disabled to your launch command.
Please report this issue at https://github.com/sgl-project/sglang/issues/new/choose
```
To work around it, add `--disable-piecewise-cuda-graph` to your launch command. When filing a bug report, please include:
To work around it, add `--cuda-graph-backend-prefill=disabled` to your launch command. When filing a bug report, please include:
1. The full error traceback
2. Model name and quantization method
3. Launch command with all arguments
@@ -228,7 +218,7 @@ The default capture schedule is auto-generated with increasing granularity:
</tbody>
</table>
For the auto-generated schedule, sizes are capped at `--piecewise-cuda-graph-max-tokens`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround.
For the auto-generated schedule, sizes are capped at `--cuda-graph-max-bs-prefill`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround.
## Compatibility
@@ -247,7 +237,7 @@ PCG is auto-disabled in the following scenarios. We are actively working on expa
- PD disaggregation
- Expert distribution recorder / EPLB
Use `--enforce-piecewise-cuda-graph` to skip all auto-disable checks (for testing/debugging only).
Use `--cuda-graph-backend-prefill=tc_piecewise` to skip all auto-disable checks (for testing/debugging only).
## Code Reference
docs/docs/advanced_features/server_arguments.mdx
-84
View File
@@ -798,12 +798,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--stream-output`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --incremental-streaming-output instead.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-streaming-session`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable streaming session mode and StreamingSession wrapper.</td>
@@ -943,12 +937,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The buckets of end-to-end request latency, specified as a list of floats.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>List[float]</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--collect-tokens-histogram`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Collect prompt/generation tokens histogram.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prompt-tokens-buckets`</td>
@@ -1522,18 +1510,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--nsa-prefill-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --dsa-prefill-backend instead.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Auto</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>flashmla_sparse</code>, <code>flashmla_kv</code>, <code>flashmla_auto</code>, <code>fa3</code>, <code>tilelang</code>, <code>aiter</code>, <code>trtllm</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--nsa-decode-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --dsa-decode-backend instead.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Auto</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>flashmla_sparse</code>, <code>flashmla_kv</code>, <code>flashmla_auto</code>, <code>fa3</code>, <code>tilelang</code>, <code>aiter</code>, <code>trtllm</code></td>
</tr>
</tbody>
</table>
@@ -1662,12 +1638,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--speculative-dflash-draft-window-size`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
</tbody>
</table>
@@ -2594,30 +2564,12 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`32`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs-decode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-max-bs-decode</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-bs`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-bs-decode</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>List[int]</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-cuda-graph`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated.</strong> Use <code>--cuda-graph-backend-decode=disabled</code> and/or <code>--cuda-graph-backend-prefill=disabled</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-breakable-cuda-graph`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-backend-prefill=breakable</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-prefill-cuda-graph`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disable the prefill-phase CUDA graph. Convenience for <code>--cuda-graph-backend-prefill=disabled</code>.</td>
@@ -2630,36 +2582,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-piecewise-cuda-graph`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-backend-prefill=disabled</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enforce-piecewise-cuda-graph`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-backend-prefill=tc_piecewise</code>. Explicitly setting the prefill backend now skips the auto-disable cascade automatically.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-tokens`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-bs-prefill</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>List[int]</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-compiler`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-tc-compiler</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>eager</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>eager</code>, <code>inductor</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-max-tokens`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-max-bs-prefill</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>4096</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-p2p-check`</td>
@@ -3485,12 +3407,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prefill-round-robin-balance`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Note: Note: --prefill-round-robin-balance is deprecated now.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
</tr>
docs/docs/advanced_features/speculative_decoding.mdx
+2 -2
View File
@@ -561,7 +561,7 @@ Relevant parameters:
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--speculative-dflash-draft-window-size</code></td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--speculative-draft-window-size</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Draft KV sliding-window size. Must be <code>&gt;= speculative-num-draft-tokens</code> when set.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
</tr>
@@ -895,7 +895,7 @@ Below is a comprehensive list of all speculative decoding parameters available i
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DFlash-only alias of <code>--speculative-num-draft-tokens</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--speculative-dflash-draft-window-size</code></td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--speculative-draft-window-size</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>int</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DFlash-only draft KV sliding-window size</td>
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx
+7 -13
View File
@@ -169,14 +169,13 @@ do
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--tokenizer-worker-num 4 \
@@ -296,7 +295,7 @@ python3 -m sglang.launch_server \
--device npu \
--quantization modelslim \
--watchdog-timeout 9000 \
--cuda-graph-bs 4 8 12 14 \
--cuda-graph-bs-decode 4 8 12 14 \
--mem-fraction-static 0.9 \
--max-running-requests 224 \
--context-length 8188 \
@@ -481,7 +480,7 @@ do
--deepep-mode low_latency \
--enable-dp-lm-head \
--moe-dense-tp 1 \
--cuda-graph-bs 2 4 6 \
--cuda-graph-bs-decode 2 4 6 \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
@@ -489,7 +488,6 @@ do
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -689,7 +687,7 @@ do
--enable-dp-attention \
--deepep-mode low_latency \
--moe-dense-tp 1 \
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
@@ -697,7 +695,6 @@ do
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -893,7 +890,7 @@ do
--deepep-mode low_latency \
--enable-dp-lm-head \
--moe-dense-tp 1 \
--cuda-graph-bs 2 4 6 \
--cuda-graph-bs-decode 2 4 6 \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
@@ -901,7 +898,6 @@ do
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -1097,7 +1093,7 @@ do
--deepep-mode low_latency \
--enable-dp-lm-head \
--moe-dense-tp 1 \
--cuda-graph-bs 2 4 6 \
--cuda-graph-bs-decode 2 4 6 \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
@@ -1105,7 +1101,6 @@ do
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -1300,14 +1295,13 @@ do
--deepep-mode low_latency \
--enable-dp-lm-head \
--moe-dense-tp 1 \
--cuda-graph-bs 2 4 6 \
--cuda-graph-bs-decode 2 4 6 \
--watchdog-timeout 9000 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash.mdx
+3 -3
View File
@@ -185,7 +185,7 @@ do
--quantization modelslim \
--enable-dp-lm-head \
--kv-cache-dtype bfloat16 \
--cuda-graph-bs 1 2 4 8 16 24 36 40 48 56 \
--cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56 \
--speculative-algorithm EAGLE \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
@@ -329,7 +329,7 @@ python3 -m sglang.launch_server \
--enable-dp-lm-head \
--kv-cache-dtype auto \
--skip-server-warmup \
--cuda-graph-bs 1 2 4 8 \
--cuda-graph-bs-decode 1 2 4 8 \
--speculative-algorithm EAGLE \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
@@ -455,7 +455,7 @@ python3 -m sglang.launch_server \
--enable-dp-lm-head \
--kv-cache-dtype auto \
--skip-server-warmup \
--cuda-graph-bs 1 2 4 8 10 \
--cuda-graph-bs-decode 1 2 4 8 10 \
--speculative-algorithm EAGLE \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1.mdx
+7 -14
View File
@@ -307,12 +307,11 @@ do
--served-model-name glm-5 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--cuda-graph-bs 1 2 3 \
--cuda-graph-bs-decode 1 2 3 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 180000 \
--tokenizer-worker-num 16 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -507,12 +506,11 @@ do
--served-model-name glm-5 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--cuda-graph-bs 1 2 3 \
--cuda-graph-bs-decode 1 2 3 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 180000 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -719,12 +717,11 @@ do
--served-model-name glm-5 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--cuda-graph-bs 1 2 3 \
--cuda-graph-bs-decode 1 2 3 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 180000 \
--tokenizer-worker-num 16 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -921,12 +918,11 @@ do
--served-model-name glm-5 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--cuda-graph-bs 1 2 3 \
--cuda-graph-bs-decode 1 2 3 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 180000 \
--tokenizer-worker-num 16 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -1120,12 +1116,11 @@ do
--served-model-name glm-5 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--cuda-graph-bs 1 2 3 \
--cuda-graph-bs-decode 1 2 3 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 180000 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -1330,12 +1325,11 @@ do
--served-model-name glm-5 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--cuda-graph-bs 1 2 3 4 5 6 \
--cuda-graph-bs-decode 1 2 3 4 5 6 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 180000 \
--tokenizer-worker-num 32 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
@@ -1540,12 +1534,11 @@ do
--served-model-name glm-5 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 \
--cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 180000 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_2.mdx
+1 -2
View File
@@ -171,12 +171,11 @@ do
--served-model-name glm-5 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--cuda-graph-max-bs 8 \
--cuda-graph-max-bs-decode 8 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 180000 \
--tokenizer-worker-num 8 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--load-balance-method round_robin \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx
+5 -5
View File
@@ -123,7 +123,7 @@ do
--dp-size 32 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs 1 \
--cuda-graph-bs-decode 1 \
--disable-radix-cache \
--speculative-algorithm EAGLE3 \
--speculative-draft-model-path $DRAFT_MODEL_PATH \
@@ -305,7 +305,7 @@ do
--sampling-backend ascend \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs 1 2 4 6 8 16 \
--cuda-graph-bs-decode 1 2 4 6 8 16 \
--speculative-algorithm EAGLE3 \
--speculative-draft-model-path $DRAFT_MODEL_PATH \
--speculative-num-steps 3 \
@@ -504,7 +504,7 @@ do
--sampling-backend ascend \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs 1 2 4 6 8 16 \
--cuda-graph-bs-decode 1 2 4 6 8 16 \
--speculative-algorithm EAGLE3 \
--speculative-draft-model-path $DRAFT_MODEL_PATH \
--speculative-num-steps 3 \
@@ -702,7 +702,7 @@ do
--sampling-backend ascend \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs 16 \
--cuda-graph-bs-decode 16 \
--reasoning-parser kimi_k2 \
--tool-call-parser kimi_k2 \
--speculative-algorithm EAGLE3 \
@@ -896,7 +896,7 @@ do
--sampling-backend ascend \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs 1 2 4 6 8 \
--cuda-graph-bs-decode 1 2 4 6 8 \
--reasoning-parser kimi_k2 \
--tool-call-parser kimi_k2 \
--speculative-algorithm EAGLE3 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx
+8 -8
View File
@@ -129,7 +129,7 @@ do
--disaggregation-transfer-backend ascend \
--disable-radix-cache \
--disable-cuda-graph \
--disable-piecewise-cuda-graph \
--cuda-graph-backend-prefill=disabled \
--dp-size 2
break
fi
@@ -159,7 +159,7 @@ do
--max-running-requests 64 \
--mem-fraction-static 0.8 \
--swa-full-tokens-ratio 0.3 \
--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \
--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
--disaggregation-transfer-backend ascend \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
@@ -314,7 +314,7 @@ do
--disaggregation-transfer-backend ascend \
--disable-radix-cache \
--disable-cuda-graph \
--disable-piecewise-cuda-graph \
--cuda-graph-backend-prefill=disabled \
--dp-size 2
break
fi
@@ -344,7 +344,7 @@ do
--max-running-requests 32 \
--mem-fraction-static 0.8 \
--swa-full-tokens-ratio 0.3 \
--cuda-graph-bs 1 2 4 8 12 16 \
--cuda-graph-bs-decode 1 2 4 8 12 16 \
--disaggregation-transfer-backend ascend \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
@@ -499,7 +499,7 @@ do
--disaggregation-transfer-backend ascend \
--disable-radix-cache \
--disable-cuda-graph \
--disable-piecewise-cuda-graph \
--cuda-graph-backend-prefill=disabled \
--dp-size 2
break
fi
@@ -529,7 +529,7 @@ do
--max-running-requests 64 \
--mem-fraction-static 0.8 \
--swa-full-tokens-ratio 0.3 \
--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \
--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
--disaggregation-transfer-backend ascend \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
@@ -684,7 +684,7 @@ do
--disaggregation-transfer-backend ascend \
--disable-radix-cache \
--disable-cuda-graph \
--disable-piecewise-cuda-graph \
--cuda-graph-backend-prefill=disabled \
--dp-size 2
break
fi
@@ -714,7 +714,7 @@ do
--max-running-requests 64 \
--mem-fraction-static 0.8 \
--swa-full-tokens-ratio 0.3 \
--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \
--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
--disaggregation-transfer-backend ascend \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx
+5 -5
View File
@@ -105,7 +105,7 @@ python3 -m sglang.launch_server \
--max-running-requests 18 \
--chunked-prefill-size -1 \
--max-prefill-tokens 32768 \
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 24 \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 24 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--quantization modelslim \
@@ -212,7 +212,7 @@ python3 -m sglang.launch_server \
--prefill-max-requests 10 \
--chunked-prefill-size 67072 \
--max-prefill-tokens 67000 \
--cuda-graph-bs 2 4 8 12 16 18 20 22 24 26 \
--cuda-graph-bs-decode 2 4 8 12 16 18 20 22 24 26 \
--moe-a2a-backend ascend_fuseep \
--deepep-mode auto \
--quantization modelslim \
@@ -324,7 +324,7 @@ python3 -m sglang.launch_server \
--prefill-max-requests 4 \
--chunked-prefill-size 160000 \
--max-prefill-tokens 80000 \
--cuda-graph-bs 2 4 6 8 \
--cuda-graph-bs-decode 2 4 6 8 \
--moe-a2a-backend ascend_fuseep \
--deepep-mode auto \
--quantization modelslim \
@@ -437,7 +437,7 @@ python3 -m sglang.launch_server \
--prefill-max-requests 3 \
--chunked-prefill-size -1 \
--max-prefill-tokens 8192 \
--cuda-graph-bs 1 2 3 4 5 6 \
--cuda-graph-bs-decode 1 2 3 4 5 6 \
--moe-a2a-backend ascend_fuseep \
--deepep-mode auto \
--quantization modelslim \
@@ -541,7 +541,7 @@ python3 -m sglang.launch_server \
--enable-prefill-delayer \
--chunked-prefill-size 196608 \
--max-prefill-tokens 8192 \
--cuda-graph-bs 1 2 4 8 12 16 20 \
--cuda-graph-bs-decode 1 2 4 8 12 16 20 \
--moe-a2a-backend ascend_fuseep \
--fuseep-mode 2 \
--quantization modelslim \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx
+2 -2
View File
@@ -101,7 +101,7 @@ python3 -m sglang.launch_server \
--enable-dp-lm-head \
--tp 16 \
--mem-fraction-static 0.78 \
--cuda-graph-bs 1 \
--cuda-graph-bs-decode 1 \
--reasoning-parser qwen3 \
--tool-call-parser qwen25
```
@@ -210,7 +210,7 @@ python3 -m sglang.launch_server \
--enable-dp-attention \
--enable-dp-lm-head \
--mem-fraction-static 0.8 \
--cuda-graph-bs 1 2 4 8 16 20 24 26 27 \
--cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27 \
--reasoning-parser qwen3 \
--tool-call-parser qwen25
```
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx
+4 -4
View File
@@ -109,7 +109,7 @@ python3 -m sglang.launch_server \
--enable-dp-attention \
--dp-size 2 \
--mem-fraction-static 0.85 \
--cuda-graph-bs 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \
--cuda-graph-bs-decode 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \
--dtype bfloat16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen
@@ -211,7 +211,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 4 \
--tp-size 2 \
--mem-fraction-static 0.87 \
--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
--dtype bfloat16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen
@@ -311,7 +311,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 4 \
--tp-size 2 \
--mem-fraction-static 0.87 \
--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
--dtype bfloat16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen
@@ -411,7 +411,7 @@ python3 -m sglang.launch_server \
--max-prefill-tokens 35000 \
--tp-size 2 \
--mem-fraction-static 0.6 \
--cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \
--cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \
--dtype bfloat16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx
+3 -3
View File
@@ -100,7 +100,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 5 \
--tp-size 16 \
--mem-fraction-static 0.72 \
--cuda-graph-bs 1 \
--cuda-graph-bs-decode 1 \
--dtype bfloat16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen
@@ -199,7 +199,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 4 \
--tp-size 4 \
--mem-fraction-static 0.845 \
--cuda-graph-bs 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
--cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
--dtype bfloat16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen
@@ -298,7 +298,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 4 \
--tp-size 4 \
--mem-fraction-static 0.845 \
--cuda-graph-bs 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
--cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
--dtype bfloat16 \
--reasoning-parser qwen3 \
--tool-call-parser qwen
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b.mdx
+9 -9
View File
@@ -111,7 +111,7 @@ python3 -m sglang.launch_server \
--trust-remote-code \
--max-running-requests 16 \
--mem-fraction-static 0.6 \
--cuda-graph-bs 2 3 4 5 6 8 10 12 14 16 \
--cuda-graph-bs-decode 2 3 4 5 6 8 10 12 14 16 \
--quantization modelslim \
--enable-multimodal \
--moe-a2a-backend deepep \
@@ -330,7 +330,7 @@ python3 -m sglang.launch_server \
--max-prefill-tokens 131072 \
--max-mamba-cache-size 320 \
--prefill-max-requests 10 \
--mamba-scheduler-strategy extra_buffer \
--mamba-radix-cache-strategy extra_buffer \
--trust-remote-code \
--max-running-requests 64 \
--mem-fraction-static 0.6 \
@@ -451,7 +451,7 @@ python3 -m sglang.launch_server \
--max-running-requests 48 \
--mem-fraction-static 0.8 \
--max-total-tokens 210000 \
--cuda-graph-bs 2 4 6 8 10 12 \
--cuda-graph-bs-decode 2 4 6 8 10 12 \
--quantization modelslim \
--enable-multimodal \
--moe-a2a-backend deepep \
@@ -569,7 +569,7 @@ python3 -m sglang.launch_server \
--max-running-requests 144 \
--mem-fraction-static 0.8 \
--max-total-tokens 635000 \
--cuda-graph-bs 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \
--cuda-graph-bs-decode 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \
--quantization modelslim \
--enable-multimodal \
--moe-a2a-backend deepep \
@@ -686,7 +686,7 @@ python3 -m sglang.launch_server \
--trust-remote-code \
--max-running-requests 160 \
--mem-fraction-static 0.8 \
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 \
--quantization modelslim \
--enable-multimodal \
--moe-a2a-backend deepep \
@@ -803,7 +803,7 @@ python3 -m sglang.launch_server \
--trust-remote-code \
--max-running-requests 432 \
--mem-fraction-static 0.8 \
--cuda-graph-bs 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \
--cuda-graph-bs-decode 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \
--quantization modelslim \
--enable-multimodal \
--moe-a2a-backend deepep \
@@ -921,7 +921,7 @@ python3 -m sglang.launch_server \
--max-running-requests 16 \
--mem-fraction-static 0.6 \
--max-total-tokens 1065000 \
--cuda-graph-bs 2 4 6 8 10 12 14 16 \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 \
--quantization modelslim \
--enable-multimodal \
--moe-a2a-backend deepep \
@@ -1039,7 +1039,7 @@ python3 -m sglang.launch_server \
--max-running-requests 32 \
--mem-fraction-static 0.6 \
--max-total-tokens 1065000 \
--cuda-graph-bs 2 4 6 8 12 14 16 \
--cuda-graph-bs-decode 2 4 6 8 12 14 16 \
--quantization modelslim \
--enable-multimodal \
--moe-a2a-backend deepep \
@@ -1147,7 +1147,7 @@ python3 -m sglang.launch_server \
--chunked-prefill-size -1 \
--max-prefill-tokens 65536 \
--max-mamba-cache-size 640 \
--mamba-scheduler-strategy extra_buffer \
--mamba-radix-cache-strategy extra_buffer \
--trust-remote-code \
--max-running-requests 128 \
--mem-fraction-static 0.6 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b.mdx
+9 -9
View File
@@ -96,7 +96,7 @@ python3 -m sglang.launch_server \
--max-running-requests 60 \
--max-mamba-cache-size 60 \
--mem-fraction-static 0.74 \
--cuda-graph-bs 2 4 8 14 16 24 26 32 36 37 40 42 44 45 46 50 52 60 \
--cuda-graph-bs-decode 2 4 8 14 16 24 26 32 36 37 40 42 44 45 46 50 52 60 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -194,7 +194,7 @@ python3 -m sglang.launch_server \
--max-running-requests 38 \
--max-mamba-cache-size 38 \
--mem-fraction-static 0.7 \
--cuda-graph-bs 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \
--cuda-graph-bs-decode 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \
--enable-prefill-delayer \
--prefill-delayer-queue-min-ratio 0.45 \
--prefill-delayer-max-delay-ms 5500 \
@@ -292,12 +292,12 @@ python3 -m sglang.launch_server \
--device npu \
--chunked-prefill-size 32768 \
--max-prefill-tokens 32768 \
--mamba-scheduler-strategy extra_buffer \
--mamba-radix-cache-strategy extra_buffer \
--trust-remote-code \
--max-running-requests 20 \
--max-mamba-cache-size 160 \
--mem-fraction-static 0.82 \
--cuda-graph-bs 1 2 5 10 15 17 19 20 \
--cuda-graph-bs-decode 1 2 5 10 15 17 19 20 \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
@@ -397,7 +397,7 @@ python3 -m sglang.launch_server \
--max-running-requests 64 \
--max-mamba-cache-size 74 \
--mem-fraction-static 0.7 \
--cuda-graph-bs 2 8 16 32 40 45 50 54 \
--cuda-graph-bs-decode 2 8 16 32 40 45 50 54 \
--enable-multimodal \
--quantization modelslim \
--mm-attention-backend ascend_attn \
@@ -491,7 +491,7 @@ python3 -m sglang.launch_server \
--max-running-requests 6 \
--max-mamba-cache-size 16 \
--mem-fraction-static 0.6 \
--cuda-graph-bs 1 2 4 5 6 \
--cuda-graph-bs-decode 1 2 4 5 6 \
--quantization modelslim \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
@@ -587,7 +587,7 @@ python3 -m sglang.launch_server \
--max-running-requests 6 \
--max-mamba-cache-size 7 \
--mem-fraction-static 0.63 \
--cuda-graph-bs 1 2 4 5 6 \
--cuda-graph-bs-decode 1 2 4 5 6 \
--enable-multimodal \
--quantization modelslim \
--mm-attention-backend ascend_attn \
@@ -682,7 +682,7 @@ python3 -m sglang.launch_server \
--max-running-requests 37 \
--max-mamba-cache-size 74 \
--mem-fraction-static 0.7 \
--cuda-graph-bs 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \
--cuda-graph-bs-decode 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \
--quantization modelslim \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
@@ -779,7 +779,7 @@ python3 -m sglang.launch_server \
--max-running-requests 28 \
--max-mamba-cache-size 50 \
--mem-fraction-static 0.7 \
--cuda-graph-bs 2 4 6 \
--cuda-graph-bs-decode 2 4 6 \
--enable-multimodal \
--quantization modelslim \
--mm-attention-backend ascend_attn \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b.mdx
+11 -11
View File
@@ -103,7 +103,7 @@ python3 -m sglang.launch_server \
--max-running-requests 120 \
--max-mamba-cache-size 120 \
--mem-fraction-static 0.85 \
--cuda-graph-bs 4 16 32 48 64 110 165 \
--cuda-graph-bs-decode 4 16 32 48 64 110 165 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -205,7 +205,7 @@ python3 -m sglang.launch_server \
--max-running-requests 42 \
--max-mamba-cache-size 42 \
--mem-fraction-static 0.75 \
--cuda-graph-bs 4 8 16 24 48 64 80 \
--cuda-graph-bs-decode 4 8 16 24 48 64 80 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -308,7 +308,7 @@ python3 -m sglang.launch_server \
--max-running-requests 3 \
--max-mamba-cache-size 3 \
--mem-fraction-static 0.9 \
--cuda-graph-bs 1 2 3 \
--cuda-graph-bs-decode 1 2 3 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -402,11 +402,11 @@ python3 -m sglang.launch_server \
--max-prefill-tokens 65536 \
--trust-remote-code \
--enable-prefill-delayer \
--mamba-scheduler-strategy extra_buffer \
--mamba-radix-cache-strategy extra_buffer \
--max-running-requests 103 \
--max-mamba-cache-size 85 \
--mem-fraction-static 0.85 \
--cuda-graph-bs 2 4 8 16 32 48 64 80 96 103 \
--cuda-graph-bs-decode 2 4 8 16 32 48 64 80 96 103 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -507,7 +507,7 @@ python3 -m sglang.launch_server \
--max-running-requests 1 \
--max-mamba-cache-size 6 \
--mem-fraction-static 0.65 \
--cuda-graph-bs 1 \
--cuda-graph-bs-decode 1 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -606,7 +606,7 @@ python3 -m sglang.launch_server \
--max-running-requests 122 \
--max-mamba-cache-size 122 \
--mem-fraction-static 0.9 \
--cuda-graph-bs 4 16 32 64 96 116 120 122 \
--cuda-graph-bs-decode 4 16 32 64 96 116 120 122 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -705,7 +705,7 @@ python3 -m sglang.launch_server \
--max-running-requests 10 \
--max-mamba-cache-size 20 \
--mem-fraction-static 0.65 \
--cuda-graph-bs 2 4 8 12 14 16 \
--cuda-graph-bs-decode 2 4 8 12 14 16 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -800,11 +800,11 @@ python3 -m sglang.launch_server \
--max-total-tokens 470784 \
--max-prefill-tokens 65536 \
--trust-remote-code \
--mamba-scheduler-strategy extra_buffer \
--mamba-radix-cache-strategy extra_buffer \
--max-running-requests 40 \
--max-mamba-cache-size 200 \
--mem-fraction-static 0.9 \
--cuda-graph-bs 2 8 16 24 32 36 40 \
--cuda-graph-bs-decode 2 8 16 24 32 36 40 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
@@ -906,7 +906,7 @@ python3 -m sglang.launch_server \
--max-running-requests 1 \
--max-mamba-cache-size 6 \
--mem-fraction-static 0.68 \
--cuda-graph-bs 1 \
--cuda-graph-bs-decode 1 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx
+3 -3
View File
@@ -95,7 +95,7 @@ python3 -m sglang.launch_server \
--chunked-prefill-size 16384 \
--tp-size 1 \
--mem-fraction-static 0.85 \
--cuda-graph-bs 8 12 24 36 48 51 55 60 63 64 66 68 70 \
--cuda-graph-bs-decode 8 12 24 36 48 51 55 60 63 64 66 68 70 \
--dtype bfloat16 \
--speculative-draft-model-quantization unquant \
--speculative-algorithm EAGLE3 \
@@ -192,7 +192,7 @@ python3 -m sglang.launch_server \
--chunked-prefill-size -1 \
--tp-size 2 \
--mem-fraction-static 0.894 \
--cuda-graph-bs 1 \
--cuda-graph-bs-decode 1 \
--dtype bfloat16 \
--speculative-draft-model-quantization unquant \
--speculative-algorithm EAGLE3 \
@@ -289,7 +289,7 @@ python3 -m sglang.launch_server \
--chunked-prefill-size -1 \
--tp-size 2 \
--mem-fraction-static 0.894 \
--cuda-graph-bs 1 5 15 16 \
--cuda-graph-bs-decode 1 5 15 16 \
--dtype bfloat16 \
--speculative-draft-model-quantization unquant \
--speculative-algorithm EAGLE3 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx
+3 -3
View File
@@ -116,7 +116,7 @@ python3 -m sglang.launch_server \
--speculative-draft-model-quantization unquant \
--chunked-prefill-size -1 \
--max-running-requests 2 \
--cuda-graph-bs 2 \
--cuda-graph-bs-decode 2 \
--mamba-ssm-dtype bfloat16 \
--speculative-draft-model-path $DRAFT_MODEL_PATH \
--reasoning-parser qwen3 \
@@ -235,7 +235,7 @@ python3 -m sglang.launch_server \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \
--cuda-graph-bs-decode 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
```
@@ -349,7 +349,7 @@ python3 -m sglang.launch_server \
--speculative-draft-model-quantization unquant \
--chunked-prefill-size -1 \
--max-running-requests 16 \
--cuda-graph-bs 2 4 8 \
--cuda-graph-bs-decode 2 4 8 \
--mamba-ssm-dtype bfloat16 \
--speculative-draft-model-path $DRAFT_MODEL_PATH \
--reasoning-parser qwen3 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ v0.5.16 or a later version.
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 20 21 22` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 20 21 22` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ recommended to use v0.5.16 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6 7 8` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6 7 8` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ recommended to use v0.5.16 or a later version.
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 24 36 40 48 56` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56` |
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx
+1 -1
View File
@@ -31,7 +31,7 @@ v0.5.16 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx
+5 -5
View File
@@ -23,7 +23,7 @@ multi-node deployment, prefill-decode disaggregation, feature configuration, and
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -220,7 +220,7 @@ python3 -m sglang.launch_server \
--mem-fraction-static 0.7 \
--port 8000 \
--served-model-name glm-5 \
--cuda-graph-bs 16 \
--cuda-graph-bs-decode 16 \
--quantization modelslim \
--speculative-draft-model-quantization unquant \
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
@@ -302,7 +302,7 @@ do
--mem-fraction-static 0.8 \
--port 8000 \
--served-model-name glm-5 \
--cuda-graph-max-bs 32 \
--cuda-graph-max-bs-decode 32 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--speculative-draft-model-quantization unquant \
@@ -421,8 +421,8 @@ do
--port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --ep-size 16 \
--mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \
--served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \
--cuda-graph-max-bs 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
--tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \
--cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
--tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \
--speculative-draft-model-quantization unquant \
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
NODE_RANK=$i
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx
+3 -3
View File
@@ -21,7 +21,7 @@ multi-node (Atlas 800I A2) PD mixed mode and speculative decoding.
| Feature | Example usage |
|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs`; e.g. `--cuda-graph-bs 4 8 16 20 24 28 32 36 40` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode`; e.g. `--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40` |
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| Reasoning Mode | `extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}` — deep chain-of-thought;<br/>`"reasoning_effort": "low"` — short thinking chain;<br/>`"reasoning_effort": "no_think"` — direct response, no thinking |
@@ -209,7 +209,7 @@ python3 -m sglang.launch_server \
--base-gpu-id 0 \
--prefill-max-requests 40 \
--max-running-requests 40 \
--cuda-graph-bs 4 8 16 20 24 28 32 36 40 \
--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \
--speculative-algorithm EAGLE \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
@@ -286,7 +286,7 @@ for i in "${!NODE_IPS[@]}"; do
--base-gpu-id 0 \
--prefill-max-requests 40 \
--max-running-requests 40 \
--cuda-graph-bs 4 8 16 20 24 28 32 36 40 \
--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \
--speculative-algorithm EAGLE \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ v0.5.16 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 24 32 48 64 96 120` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 4 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 5 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.16 or a later version.
| Expert Parallelism | `--moe-a2a-backend deepep \`<br/>`--deepep-mode low_latency` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32` |
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--enable-multi-layer-eagle` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx
+1 -1
View File
@@ -28,7 +28,7 @@ v0.5.16 or a later version.
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend ascend_fuseep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 8 16 24 32 48 64 80` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 8 16 24 32 48 64 80` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx
+1 -1
View File
@@ -30,7 +30,7 @@ v0.5.16 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 94208` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.16 or a later version.
| Tensor Parallelism | `--tp-size 2` |
| Data Parallelism | `--dp-size 2` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx
+1 -1
View File
@@ -24,7 +24,7 @@ version.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 4` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 4 8 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 4 8 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx
+1 -1
View File
@@ -28,7 +28,7 @@ use v0.5.16 or a later version.
| Data Parallelism | `--dp-size 8` |
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ v0.5.16 or a later version.
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 8 16 32 48` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 8 16 32 48` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ version**.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 2` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 16 24 32 48 64 80 96 112 120` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx
+2 -2
View File
@@ -28,7 +28,7 @@ source.
| Expert Parallelism | `--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -215,7 +215,7 @@ do
--enable-dp-attention --dp-size 4 --enable-dp-lm-head \
--mem-fraction-static 0.8 \
--chunked-prefill-size 8192 \
--cuda-graph-bs 16 \
--cuda-graph-bs-decode 16 \
--disable-radix-cache \
--max-running-requests 64 \
--host 0.0.0.0 \
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx
+1 -1
View File
@@ -26,7 +26,7 @@ version.
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 6 9 10 15 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 6 9 10 15 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx
+1 -1
View File
@@ -27,7 +27,7 @@ v0.5.16 or a later version.
| Data Parallelism | `--dp-size 2` |
| Expert Parallelism | `--ep-size 4 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 8` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 8` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
docs/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning.mdx
+2 -2
View File
@@ -755,9 +755,9 @@ see [Quantization on Ascend](/docs/hardware-platforms/ascend-npus/optimization/q
Breaks large prefill computations into smaller, fixed-size chunks for better scheduling interleaving with decode batches. This prevents long prefill requests from blocking decode latency. Use `-1` to disable chunked prefill on dedicated prefill nodes. Has partial compatibility with most features because chunk boundaries introduce scheduling complexity.
### NPU Graph (`--cuda-graph-bs`)
### NPU Graph (`--cuda-graph-bs-decode`)
Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes.
Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs-decode`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes.
<Note>
`--enable-torch-compile` is incompatible with NPU Graph. When torch.compile is enabled, NPU Graph must be disabled via `--disable-cuda-graph`.
docs/docs/hardware-platforms/ascend-npus/reference/support_features.mdx
+6 -18
View File
@@ -816,12 +816,6 @@ click [Server Arguments](../../../advanced_features/server_arguments).
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>List[float]</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--collect-tokens-histogram`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag<br/> (set to enable)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prompt-tokens-buckets`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
@@ -1498,7 +1492,7 @@ non-default speculative acceptance thresholds or deterministic inference.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--speculative-dflash-draft-window-size`<br/>`--speculative-draft-window-size`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--speculative-draft-window-size`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
@@ -2073,7 +2067,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-backend-prefill`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`disabled`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`disabled`, `tc_piecewise`<br/> (`tc_piecewise` currently supports Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
@@ -2113,7 +2107,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-bs`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-bs-decode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>List[int]</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
@@ -2257,19 +2251,13 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enforce-piecewise-cuda-graph`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag<br/> (set to enable); <br/> Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-tokens`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-bs-prefill`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: JSON<br/> list</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-compiler`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-tc-compiler`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`eager`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`eager`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
@@ -2281,7 +2269,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-max-tokens`</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs-prefill`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
docs/docs/sglang-diffusion/disaggregation.mdx
+1 -1
View File
@@ -223,7 +223,7 @@ sglang serve \
--model-path ./zai-org/GLM-Image/vision_language_encoder/ \
--tokenizer-path ./zai-org/GLM-Image/processor/ \
--enable-multimodal \
--cuda-graph-max-bs 28 \
--cuda-graph-max-bs-decode 28 \
--device npu \
--attention-backend ascend \
--disable-fast-image-processor \
docs/docs/sglang-diffusion/models_with_ar.mdx
+1 -1
View File
@@ -124,7 +124,7 @@ export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199"
export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399"
sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \
--tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal \
--cuda-graph-bs 1 --device npu --attention-backend ascend --image-processor-backend pil \
--cuda-graph-bs-decode 1 --device npu --attention-backend ascend --image-processor-backend pil \
--tp-size 4 --port ${PORT} --mem-fraction-static 0.4
```
Second terminal with diffusion server:
docs/src/snippets/configs/Qwen/qwen3.8-27b.jsx
+6 -6
View File
@@ -589,7 +589,7 @@ export const config = {
// RTX 5090 32GB. NVFP4 is the only checkpoint that fits (FP8 does not
// boot — total_rest_memory negative at every mem-fraction, measured —
// and BF16 does not fit). Published operating point is ONE request in
// flight; --cuda-graph-max-bs 1 also protects the token pool (default
// flight; --cuda-graph-max-bs-decode 1 also protects the token pool (default
// capture set costs 39,247 -> 37,347 and K 8 -> 7). The `warn` below
// carries the user-facing guidance for raising concurrency.
match: { hw: "rtx5090", variant: "default", quant: "nvfp4-bf16-head", nodes: "single" },
@@ -598,7 +598,7 @@ export const config = {
// multi-user deployment unaware.
warn:
"This recipe serves ONE request at a time: --max-running-requests 1 " +
"and --cuda-graph-max-bs 1 pin it to the validated single-stream " +
"and --cuda-graph-max-bs-decode 1 pin it to the validated single-stream " +
"envelope. To handle more concurrent requests, raise both flags " +
"together and re-derive --mamba-full-memory-ratio (and mem-fraction) " +
"with the [Mamba ratio calculator](#mamba-ratio-calculator) — on this " +
@@ -611,7 +611,7 @@ export const config = {
"--mem-fraction-static 0.9",
"--attention-backend flashinfer",
"--max-running-requests 1",
"--cuda-graph-max-bs 1",
"--cuda-graph-max-bs-decode 1",
"--reasoning-parser qwen3",
"--tool-call-parser qwen3_coder",
"--host {{HOST_IP}}",
@@ -624,7 +624,7 @@ export const config = {
// RTX 5090 32GB. NVFP4 is the only checkpoint that fits (FP8 does not
// boot — total_rest_memory negative at every mem-fraction, measured —
// and BF16 does not fit). Published operating point is ONE request in
// flight; --cuda-graph-max-bs 1 also protects the token pool (default
// flight; --cuda-graph-max-bs-decode 1 also protects the token pool (default
// capture set costs 39,247 -> 37,347 and K 8 -> 7). The `warn` below
// carries the user-facing guidance for raising concurrency.
match: { hw: "rtx5090", variant: "default", quant: "nvfp4-fp4-head", nodes: "single" },
@@ -633,7 +633,7 @@ export const config = {
// multi-user deployment unaware.
warn:
"This recipe serves ONE request at a time: --max-running-requests 1 " +
"and --cuda-graph-max-bs 1 pin it to the validated single-stream " +
"and --cuda-graph-max-bs-decode 1 pin it to the validated single-stream " +
"envelope. To handle more concurrent requests, raise both flags " +
"together and re-derive --mamba-full-memory-ratio (and mem-fraction) " +
"with the [Mamba ratio calculator](#mamba-ratio-calculator) — on this " +
@@ -646,7 +646,7 @@ export const config = {
"--mem-fraction-static 0.9",
"--attention-backend flashinfer",
"--max-running-requests 1",
"--cuda-graph-max-bs 1",
"--cuda-graph-max-bs-decode 1",
"--reasoning-parser qwen3",
"--tool-call-parser qwen3_coder",
"--host {{HOST_IP}}",
docs/src/snippets/configs/Qwen/qwen3.8.jsx
+2 -2
View File
@@ -233,8 +233,8 @@ export const config = {
// ----- Card: "PD Disaggregation" -----
// Role flags follow the P/D bundle's own prefill and decode workers. Two
// flags those recipes carry are deliberately not emitted:
// --prefill-round-robin-balance is a DeprecatedAction on current SGLang and
// does nothing, and --mamba-track-interval is context-dependent (the source
// --prefill-round-robin-balance no longer exists on current SGLang (it was
// a deprecated no-op), and --mamba-track-interval is context-dependent (the source
// recipes set it equal to their context cap) so a fixed value here would be
// wrong for cells serving the native window.
pdDisagg: {
docs/src/snippets/configs/inclusionAI/ling-3.0-flash.jsx
+5 -5
View File
@@ -114,7 +114,7 @@ sgl-eval run gsm8k \\
hicache: {
defaultBackend: "mooncake",
requiredFlags: [
"--mamba-scheduler-strategy extra_buffer",
"--mamba-radix-cache-strategy extra_buffer",
"--enable-cache-report",
],
backends: [
@@ -618,7 +618,7 @@ sgl-eval run gsm8k \\
"--hicache-storage-backend mooncake",
"--hicache-io-backend direct",
"--hicache-mem-layout page_first_direct",
"--mamba-scheduler-strategy extra_buffer",
"--mamba-radix-cache-strategy extra_buffer",
"--enable-cache-report",
"--hicache-storage-prefetch-policy wait_complete",
"--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'",
@@ -647,7 +647,7 @@ sgl-eval run gsm8k \\
"--hicache-storage-backend mooncake",
"--hicache-io-backend direct",
"--hicache-mem-layout page_first_direct",
"--mamba-scheduler-strategy extra_buffer",
"--mamba-radix-cache-strategy extra_buffer",
"--enable-cache-report",
"--hicache-storage-prefetch-policy wait_complete",
"--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'",
@@ -675,7 +675,7 @@ sgl-eval run gsm8k \\
"--hicache-storage-backend mooncake",
"--hicache-io-backend direct",
"--hicache-mem-layout page_first_direct",
"--mamba-scheduler-strategy extra_buffer",
"--mamba-radix-cache-strategy extra_buffer",
"--enable-cache-report",
"--hicache-storage-prefetch-policy wait_complete",
"--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'",
@@ -704,7 +704,7 @@ sgl-eval run gsm8k \\
"--hicache-storage-backend mooncake",
"--hicache-io-backend direct",
"--hicache-mem-layout page_first_direct",
"--mamba-scheduler-strategy extra_buffer",
"--mamba-radix-cache-strategy extra_buffer",
"--enable-cache-report",
"--hicache-storage-prefetch-policy wait_complete",
"--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'",
docs/src/snippets/configs/meituan-longcat/longcat-2.0.jsx
+4 -4
View File
@@ -134,7 +134,7 @@ sgl-eval run gsm8k \\
"--max-running-requests 64",
"--mem-fraction-static 0.92",
"--chunked-prefill-size 2048",
"--nsa-prefill-backend fa3",
"--dsa-prefill-backend fa3",
"--kv-cache-dtype bfloat16",
"--model-loader-extra-config '{\"enable_multithread_load\":true,\"num_threads\":12}'",
"--host {{HOST_IP}}",
@@ -154,7 +154,7 @@ sgl-eval run gsm8k \\
"--max-running-requests 64",
"--mem-fraction-static 0.92",
"--chunked-prefill-size 2048",
"--nsa-prefill-backend fa3",
"--dsa-prefill-backend fa3",
"--kv-cache-dtype bfloat16",
"--model-loader-extra-config '{\"enable_multithread_load\":true,\"num_threads\":12}'",
"--host {{HOST_IP}}",
@@ -173,7 +173,7 @@ sgl-eval run gsm8k \\
"--max-running-requests 64",
"--mem-fraction-static 0.92",
"--chunked-prefill-size 2048",
"--nsa-prefill-backend fa3",
"--dsa-prefill-backend fa3",
"--kv-cache-dtype bfloat16",
"--model-loader-extra-config '{\"enable_multithread_load\":true,\"num_threads\":12}'",
"--host {{HOST_IP}}",
@@ -192,7 +192,7 @@ sgl-eval run gsm8k \\
"--max-running-requests 64",
"--mem-fraction-static 0.92",
"--chunked-prefill-size 2048",
"--nsa-prefill-backend fa3",
"--dsa-prefill-backend fa3",
"--kv-cache-dtype bfloat16",
"--model-loader-extra-config '{\"enable_multithread_load\":true,\"num_threads\":12}'",
"--host {{HOST_IP}}",
docs/src/snippets/configs/moonshotai/kimi-k3.jsx
+5 -5
View File
@@ -1183,7 +1183,7 @@ export const config = {
"--kv-cache-dtype fp8_e4m3",
"--dtype bfloat16",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--reasoning-parser kimi_k3",
"--tool-call-parser kimi_k3",
"--host {{HOST_IP}}",
@@ -1210,7 +1210,7 @@ export const config = {
"--kv-cache-dtype fp8_e4m3",
"--dtype bfloat16",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--reasoning-parser kimi_k3",
"--tool-call-parser kimi_k3",
"--host {{HOST_IP}}",
@@ -1959,7 +1959,7 @@ export const config = {
"--kv-cache-dtype fp8_e4m3",
"--dtype bfloat16",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--reasoning-parser kimi_k3",
"--tool-call-parser kimi_k3",
"--disaggregation-mode decode",
@@ -1987,7 +1987,7 @@ export const config = {
"--kv-cache-dtype fp8_e4m3",
"--dtype bfloat16",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--reasoning-parser kimi_k3",
"--tool-call-parser kimi_k3",
"--disaggregation-mode decode",
@@ -2379,7 +2379,7 @@ export const config = {
"--enable-dp-lm-head",
"--mem-fraction-static 0.78",
"--chunked-prefill-size 16384",
"--cuda-graph-bs 2 4 8 16",
"--cuda-graph-bs-decode 2 4 8 16",
"--max-running-requests 64",
"--max-mamba-cache-size 64",
"--moe-a2a-backend deepep",
docs/src/snippets/configs/openbmb/minicpm5-2b-benchmarks.jsx
+1 -1
View File
@@ -4,7 +4,7 @@
// RTX 5090 32GB (`zijiexia-5090-1gpu`, `lmsysorg/sglang:dev` @ 30705c004c) with
// `python3 -m sglang.bench_serving`, random 1024/1024, against the exact cell
// command on this page — including `--mem-fraction-static 0.75
// --cuda-graph-max-bs 128`, which is what keeps concurrency 64 and 128
// --cuda-graph-max-bs-decode 128`, which is what keeps concurrency 64 and 128
// graph-backed (see the model page's Configuration Tips).
//
// `tokens_per_sec_per_gpu` is total (input+output) per GPU = the measured
docs/src/snippets/configs/openbmb/minicpm5-2b.jsx
+2 -2
View File
@@ -206,7 +206,7 @@ export const config = {
"--reasoning-parser qwen3",
"--tool-call-parser minicpm5",
"--mem-fraction-static 0.75",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -225,7 +225,7 @@ export const config = {
"--reasoning-parser qwen3",
"--tool-call-parser minicpm5",
"--mem-fraction-static 0.30",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
docs/src/snippets/configs/zai-org/glm-5.2.jsx
+15 -15
View File
@@ -772,7 +772,7 @@ sgl-eval run aime25 \\
"--chunked-prefill-size 8192",
"--mem-fraction-static 0.85",
"--max-running-requests 16",
"--cuda-graph-max-bs 16",
"--cuda-graph-max-bs-decode 16",
"--max-prefill-tokens 8192",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -837,7 +837,7 @@ sgl-eval run aime25 \\
"--chunked-prefill-size 8192",
"--mem-fraction-static 0.85",
"--max-running-requests 16",
"--cuda-graph-max-bs 16",
"--cuda-graph-max-bs-decode 16",
"--max-prefill-tokens 8192",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -927,7 +927,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -944,7 +944,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -978,7 +978,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -995,7 +995,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1043,7 +1043,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1062,7 +1062,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1094,7 +1094,7 @@ sgl-eval run aime25 \\
"--speculative-num-draft-tokens 4",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--cuda-graph-max-bs 160",
"--cuda-graph-max-bs-decode 160",
"--max-running-requests 160",
"--watchdog-timeout 1800",
"--host {{HOST_IP}}",
@@ -1128,7 +1128,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1145,7 +1145,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1179,7 +1179,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1196,7 +1196,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1230,7 +1230,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1247,7 +1247,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
docs/src/snippets/configs/zai-org/glm-5.3-flash.jsx
+4 -4
View File
@@ -416,7 +416,7 @@ sgl-eval run gsm8k \\
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -472,7 +472,7 @@ sgl-eval run gsm8k \\
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -518,7 +518,7 @@ sgl-eval run gsm8k \\
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
@@ -564,7 +564,7 @@ sgl-eval run gsm8k \\
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 32",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
docs/src/snippets/configs/zai-org/glm-5.3.jsx
+12 -12
View File
@@ -716,7 +716,7 @@ sgl-eval run aime25 \\
"--chunked-prefill-size 8192",
"--mem-fraction-static 0.85",
"--max-running-requests 16",
"--cuda-graph-max-bs 16",
"--cuda-graph-max-bs-decode 16",
"--max-prefill-tokens 8192",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -758,7 +758,7 @@ sgl-eval run aime25 \\
"--chunked-prefill-size 8192",
"--mem-fraction-static 0.85",
"--max-running-requests 16",
"--cuda-graph-max-bs 16",
"--cuda-graph-max-bs-decode 16",
"--max-prefill-tokens 8192",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -825,7 +825,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -842,7 +842,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -876,7 +876,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -893,7 +893,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -927,7 +927,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -944,7 +944,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -978,7 +978,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -995,7 +995,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1029,7 +1029,7 @@ sgl-eval run aime25 \\
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 128",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
@@ -1046,7 +1046,7 @@ sgl-eval run aime25 \\
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--mem-fraction-static 0.85",
"--cuda-graph-max-bs 256",
"--cuda-graph-max-bs-decode 256",
"--max-running-requests 256",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",