[Config] Retire get_global_server_args, and clear the deprecated flags that have a replacement (#38375)
This commit is contained in:
Cheng Wan
@@ -61,14 +61,17 @@ SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-8B-Instruct
|
||||
```
|
||||
Or you can run CUDA Graph for ViT together with Piecewise CUDA Graph feature by both setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` and setting `--enable-piecewise-cuda-graph`, for example:
|
||||
To run the ViT graph together with Piecewise CUDA Graph over the decoder's
|
||||
prefill, ask for the prefill backend explicitly. Qwen3-VL is on neither
|
||||
multimodal allowlist, so the default backend resolves prefill to `disabled` for
|
||||
it; naming a backend locks the choice and skips that rule:
|
||||
```shell Command
|
||||
SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-8B-Instruct \
|
||||
--piecewise-cuda-graph-max-tokens 4096 \
|
||||
--enable-piecewise-cuda-graph \
|
||||
--piecewise-cuda-graph-compiler eager
|
||||
--cuda-graph-backend-prefill tc_piecewise \
|
||||
--cuda-graph-max-bs-prefill 4096 \
|
||||
--cuda-graph-tc-compiler eager
|
||||
```
|
||||
|
||||
## Known supported models
|
||||
|
||||
@@ -568,7 +568,7 @@ python3 -m sglang.launch_server \
|
||||
--disaggregation-bootstrap-port 8996 \
|
||||
--base-gpu-id 8 \
|
||||
--disable-radix-cache \
|
||||
--cuda-graph-bs 1 2 4 8 10 12 14 16 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 10 12 14 16 \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
|
||||
--enable-multi-layer-eagle \
|
||||
@@ -681,9 +681,8 @@ do
|
||||
--port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \
|
||||
--mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \
|
||||
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
|
||||
--prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
|
||||
--load-balance-method round_robin
|
||||
NODE_RANK=$i
|
||||
break
|
||||
|
||||
@@ -10,7 +10,7 @@ Standard CUDA graphs capture the entire model forward pass as a single graph. Th
|
||||
|
||||
Piecewise CUDA Graph (PCG) solves this by splitting the model's computation graph into pieces (roughly one per layer) at "split points" (e.g., MoE dispatch ops). Each piece is captured as a separate CUDA graph for a set of pre-defined token lengths. At runtime, the input is padded to the nearest captured size, and each piece is replayed. This eliminates kernel launch overhead for prefill/extend while still supporting dynamic shapes.
|
||||
|
||||
Recently we **enabled PCG by default**, which means that the old `--enable-piecewise-cuda-graph` flag is deprecated. Use `--disable-piecewise-cuda-graph` to turn it off.
|
||||
PCG is **enabled by default**. Pass `--cuda-graph-backend-prefill=disabled` to turn it off.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -26,7 +26,7 @@ python3 -m sglang.launch_server \
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disable-piecewise-cuda-graph
|
||||
--cuda-graph-backend-prefill=disabled
|
||||
```
|
||||
|
||||
### Custom capture sizes
|
||||
@@ -34,7 +34,7 @@ python3 -m sglang.launch_server \
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--piecewise-cuda-graph-max-tokens 2048
|
||||
--cuda-graph-max-bs-prefill 2048
|
||||
```
|
||||
|
||||
### Server Args
|
||||
@@ -54,41 +54,31 @@ python3 -m sglang.launch_server \
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--disable-piecewise-cuda-graph</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Disable PCG for extend/prefill.</td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--cuda-graph-backend-prefill</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend for the prefill phase. Choices: <code>full</code>, <code>breakable</code>, <code>tc_piecewise</code>, <code>disabled</code>. Pass <code>disabled</code> to turn PCG off for extend/prefill, or <code>tc_piecewise</code> to force it on, skipping all auto-disable conditions (testing only).</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--enforce-piecewise-cuda-graph</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Force-enable PCG, skipping all auto-disable conditions. For testing only.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-max-tokens</code></td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--cuda-graph-max-bs-prefill</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Maximum token count to capture. Defaults to <code>chunked_prefill_size</code> (non-MLA) or <code>2048</code> (MLA).</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-tokens</code></td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--cuda-graph-bs-prefill</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Explicit list of token lengths to capture. Auto-generated if not set.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-compiler</code></td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--cuda-graph-tc-compiler</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>"eager"</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Compiler backend for the captured subgraphs. Choices: <code>eager</code>, <code>inductor</code>.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><del><code>--enable-piecewise-cuda-graph</code></del></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Deprecated.</strong> PCG is now enabled by default. Use <code>--enforce-piecewise-cuda-graph</code> to skip auto-disable conditions.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Bug Report
|
||||
|
||||
PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--disable-piecewise-cuda-graph` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature.
|
||||
PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--cuda-graph-backend-prefill=disabled` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature.
|
||||
|
||||
### For Users
|
||||
|
||||
@@ -96,11 +86,11 @@ If you see an error message like the following during server startup, it is a PC
|
||||
|
||||
```
|
||||
Piecewise CUDA Graph is enabled by default as an experimental feature.
|
||||
To work around this error, add --disable-piecewise-cuda-graph to your launch command.
|
||||
To work around this error, add --cuda-graph-backend-prefill=disabled to your launch command.
|
||||
Please report this issue at https://github.com/sgl-project/sglang/issues/new/choose
|
||||
```
|
||||
|
||||
To work around it, add `--disable-piecewise-cuda-graph` to your launch command. When filing a bug report, please include:
|
||||
To work around it, add `--cuda-graph-backend-prefill=disabled` to your launch command. When filing a bug report, please include:
|
||||
1. The full error traceback
|
||||
2. Model name and quantization method
|
||||
3. Launch command with all arguments
|
||||
@@ -228,7 +218,7 @@ The default capture schedule is auto-generated with increasing granularity:
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
For the auto-generated schedule, sizes are capped at `--piecewise-cuda-graph-max-tokens`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround.
|
||||
For the auto-generated schedule, sizes are capped at `--cuda-graph-max-bs-prefill`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround.
|
||||
|
||||
## Compatibility
|
||||
|
||||
@@ -247,7 +237,7 @@ PCG is auto-disabled in the following scenarios. We are actively working on expa
|
||||
- PD disaggregation
|
||||
- Expert distribution recorder / EPLB
|
||||
|
||||
Use `--enforce-piecewise-cuda-graph` to skip all auto-disable checks (for testing/debugging only).
|
||||
Use `--cuda-graph-backend-prefill=tc_piecewise` to skip all auto-disable checks (for testing/debugging only).
|
||||
|
||||
## Code Reference
|
||||
|
||||
|
||||
@@ -798,12 +798,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--stream-output`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --incremental-streaming-output instead.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-streaming-session`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable streaming session mode and StreamingSession wrapper.</td>
|
||||
@@ -943,12 +937,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The buckets of end-to-end request latency, specified as a list of floats.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>List[float]</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--collect-tokens-histogram`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Collect prompt/generation tokens histogram.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prompt-tokens-buckets`</td>
|
||||
@@ -1522,18 +1510,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--nsa-prefill-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --dsa-prefill-backend instead.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Auto</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>flashmla_sparse</code>, <code>flashmla_kv</code>, <code>flashmla_auto</code>, <code>fa3</code>, <code>tilelang</code>, <code>aiter</code>, <code>trtllm</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--nsa-decode-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --dsa-decode-backend instead.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Auto</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>flashmla_sparse</code>, <code>flashmla_kv</code>, <code>flashmla_auto</code>, <code>fa3</code>, <code>tilelang</code>, <code>aiter</code>, <code>trtllm</code></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -1662,12 +1638,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--speculative-dflash-draft-window-size`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -2594,30 +2564,12 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`32`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs-decode`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-max-bs-decode</code>.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-bs`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-bs-decode</code>.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>List[int]</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-cuda-graph`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated.</strong> Use <code>--cuda-graph-backend-decode=disabled</code> and/or <code>--cuda-graph-backend-prefill=disabled</code>.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-breakable-cuda-graph`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-backend-prefill=breakable</code>.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-prefill-cuda-graph`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disable the prefill-phase CUDA graph. Convenience for <code>--cuda-graph-backend-prefill=disabled</code>.</td>
|
||||
@@ -2630,36 +2582,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-piecewise-cuda-graph`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-backend-prefill=disabled</code>.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enforce-piecewise-cuda-graph`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-backend-prefill=tc_piecewise</code>. Explicitly setting the prefill backend now skips the auto-disable cascade automatically.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-tokens`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-bs-prefill</code>.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>List[int]</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-compiler`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-tc-compiler</code>.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>eager</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>eager</code>, <code>inductor</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-max-tokens`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated alias</strong> for <code>--cuda-graph-max-bs-prefill</code>.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>4096</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-p2p-check`</td>
|
||||
@@ -3485,12 +3407,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
</thead>
|
||||
<tbody>
|
||||
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prefill-round-robin-balance`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Note: Note: --prefill-round-robin-balance is deprecated now.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -561,7 +561,7 @@ Relevant parameters:
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--speculative-dflash-draft-window-size</code></td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--speculative-draft-window-size</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Draft KV sliding-window size. Must be <code>>= speculative-num-draft-tokens</code> when set.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
|
||||
</tr>
|
||||
@@ -895,7 +895,7 @@ Below is a comprehensive list of all speculative decoding parameters available i
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DFlash-only alias of <code>--speculative-num-draft-tokens</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--speculative-dflash-draft-window-size</code></td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--speculative-draft-window-size</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>int</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DFlash-only draft KV sliding-window size</td>
|
||||
|
||||
+7
-13
@@ -169,14 +169,13 @@ do
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 2 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 3 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--tokenizer-worker-num 4 \
|
||||
@@ -296,7 +295,7 @@ python3 -m sglang.launch_server \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--watchdog-timeout 9000 \
|
||||
--cuda-graph-bs 4 8 12 14 \
|
||||
--cuda-graph-bs-decode 4 8 12 14 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--max-running-requests 224 \
|
||||
--context-length 8188 \
|
||||
@@ -481,7 +480,7 @@ do
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--moe-dense-tp 1 \
|
||||
--cuda-graph-bs 2 4 6 \
|
||||
--cuda-graph-bs-decode 2 4 6 \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
@@ -489,7 +488,6 @@ do
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -689,7 +687,7 @@ do
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
--moe-dense-tp 1 \
|
||||
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
@@ -697,7 +695,6 @@ do
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 2 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -893,7 +890,7 @@ do
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--moe-dense-tp 1 \
|
||||
--cuda-graph-bs 2 4 6 \
|
||||
--cuda-graph-bs-decode 2 4 6 \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
@@ -901,7 +898,6 @@ do
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -1097,7 +1093,7 @@ do
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--moe-dense-tp 1 \
|
||||
--cuda-graph-bs 2 4 6 \
|
||||
--cuda-graph-bs-decode 2 4 6 \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
@@ -1105,7 +1101,6 @@ do
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -1300,14 +1295,13 @@ do
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--moe-dense-tp 1 \
|
||||
--cuda-graph-bs 2 4 6 \
|
||||
--cuda-graph-bs-decode 2 4 6 \
|
||||
--watchdog-timeout 9000 \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
|
||||
+3
-3
@@ -185,7 +185,7 @@ do
|
||||
--quantization modelslim \
|
||||
--enable-dp-lm-head \
|
||||
--kv-cache-dtype bfloat16 \
|
||||
--cuda-graph-bs 1 2 4 8 16 24 36 40 48 56 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 2 \
|
||||
--speculative-eagle-topk 1 \
|
||||
@@ -329,7 +329,7 @@ python3 -m sglang.launch_server \
|
||||
--enable-dp-lm-head \
|
||||
--kv-cache-dtype auto \
|
||||
--skip-server-warmup \
|
||||
--cuda-graph-bs 1 2 4 8 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 2 \
|
||||
--speculative-eagle-topk 1 \
|
||||
@@ -455,7 +455,7 @@ python3 -m sglang.launch_server \
|
||||
--enable-dp-lm-head \
|
||||
--kv-cache-dtype auto \
|
||||
--skip-server-warmup \
|
||||
--cuda-graph-bs 1 2 4 8 10 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 10 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 2 \
|
||||
--speculative-eagle-topk 1 \
|
||||
|
||||
+7
-14
@@ -307,12 +307,11 @@ do
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--cuda-graph-bs 1 2 3 \
|
||||
--cuda-graph-bs-decode 1 2 3 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 180000 \
|
||||
--tokenizer-worker-num 16 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -507,12 +506,11 @@ do
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--cuda-graph-bs 1 2 3 \
|
||||
--cuda-graph-bs-decode 1 2 3 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 180000 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -719,12 +717,11 @@ do
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--cuda-graph-bs 1 2 3 \
|
||||
--cuda-graph-bs-decode 1 2 3 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 180000 \
|
||||
--tokenizer-worker-num 16 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -921,12 +918,11 @@ do
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--cuda-graph-bs 1 2 3 \
|
||||
--cuda-graph-bs-decode 1 2 3 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 180000 \
|
||||
--tokenizer-worker-num 16 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -1120,12 +1116,11 @@ do
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--cuda-graph-bs 1 2 3 \
|
||||
--cuda-graph-bs-decode 1 2 3 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 180000 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -1330,12 +1325,11 @@ do
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--cuda-graph-bs 1 2 3 4 5 6 \
|
||||
--cuda-graph-bs-decode 1 2 3 4 5 6 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 180000 \
|
||||
--tokenizer-worker-num 32 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
@@ -1540,12 +1534,11 @@ do
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 \
|
||||
--cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 180000 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
|
||||
+1
-2
@@ -171,12 +171,11 @@ do
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--cuda-graph-max-bs 8 \
|
||||
--cuda-graph-max-bs-decode 8 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 180000 \
|
||||
--tokenizer-worker-num 8 \
|
||||
--prefill-round-robin-balance \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--load-balance-method round_robin \
|
||||
|
||||
+5
-5
@@ -123,7 +123,7 @@ do
|
||||
--dp-size 32 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--cuda-graph-bs 1 \
|
||||
--cuda-graph-bs-decode 1 \
|
||||
--disable-radix-cache \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path $DRAFT_MODEL_PATH \
|
||||
@@ -305,7 +305,7 @@ do
|
||||
--sampling-backend ascend \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--cuda-graph-bs 1 2 4 6 8 16 \
|
||||
--cuda-graph-bs-decode 1 2 4 6 8 16 \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path $DRAFT_MODEL_PATH \
|
||||
--speculative-num-steps 3 \
|
||||
@@ -504,7 +504,7 @@ do
|
||||
--sampling-backend ascend \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--cuda-graph-bs 1 2 4 6 8 16 \
|
||||
--cuda-graph-bs-decode 1 2 4 6 8 16 \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path $DRAFT_MODEL_PATH \
|
||||
--speculative-num-steps 3 \
|
||||
@@ -702,7 +702,7 @@ do
|
||||
--sampling-backend ascend \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--cuda-graph-bs 16 \
|
||||
--cuda-graph-bs-decode 16 \
|
||||
--reasoning-parser kimi_k2 \
|
||||
--tool-call-parser kimi_k2 \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
@@ -896,7 +896,7 @@ do
|
||||
--sampling-backend ascend \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--cuda-graph-bs 1 2 4 6 8 \
|
||||
--cuda-graph-bs-decode 1 2 4 6 8 \
|
||||
--reasoning-parser kimi_k2 \
|
||||
--tool-call-parser kimi_k2 \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
|
||||
+8
-8
@@ -129,7 +129,7 @@ do
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disable-radix-cache \
|
||||
--disable-cuda-graph \
|
||||
--disable-piecewise-cuda-graph \
|
||||
--cuda-graph-backend-prefill=disabled \
|
||||
--dp-size 2
|
||||
break
|
||||
fi
|
||||
@@ -159,7 +159,7 @@ do
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--swa-full-tokens-ratio 0.3 \
|
||||
--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
@@ -314,7 +314,7 @@ do
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disable-radix-cache \
|
||||
--disable-cuda-graph \
|
||||
--disable-piecewise-cuda-graph \
|
||||
--cuda-graph-backend-prefill=disabled \
|
||||
--dp-size 2
|
||||
break
|
||||
fi
|
||||
@@ -344,7 +344,7 @@ do
|
||||
--max-running-requests 32 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--swa-full-tokens-ratio 0.3 \
|
||||
--cuda-graph-bs 1 2 4 8 12 16 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 12 16 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
@@ -499,7 +499,7 @@ do
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disable-radix-cache \
|
||||
--disable-cuda-graph \
|
||||
--disable-piecewise-cuda-graph \
|
||||
--cuda-graph-backend-prefill=disabled \
|
||||
--dp-size 2
|
||||
break
|
||||
fi
|
||||
@@ -529,7 +529,7 @@ do
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--swa-full-tokens-ratio 0.3 \
|
||||
--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
@@ -684,7 +684,7 @@ do
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disable-radix-cache \
|
||||
--disable-cuda-graph \
|
||||
--disable-piecewise-cuda-graph \
|
||||
--cuda-graph-backend-prefill=disabled \
|
||||
--dp-size 2
|
||||
break
|
||||
fi
|
||||
@@ -714,7 +714,7 @@ do
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--swa-full-tokens-ratio 0.3 \
|
||||
--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
|
||||
+5
-5
@@ -105,7 +105,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 18 \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 32768 \
|
||||
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 24 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 24 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--quantization modelslim \
|
||||
@@ -212,7 +212,7 @@ python3 -m sglang.launch_server \
|
||||
--prefill-max-requests 10 \
|
||||
--chunked-prefill-size 67072 \
|
||||
--max-prefill-tokens 67000 \
|
||||
--cuda-graph-bs 2 4 8 12 16 18 20 22 24 26 \
|
||||
--cuda-graph-bs-decode 2 4 8 12 16 18 20 22 24 26 \
|
||||
--moe-a2a-backend ascend_fuseep \
|
||||
--deepep-mode auto \
|
||||
--quantization modelslim \
|
||||
@@ -324,7 +324,7 @@ python3 -m sglang.launch_server \
|
||||
--prefill-max-requests 4 \
|
||||
--chunked-prefill-size 160000 \
|
||||
--max-prefill-tokens 80000 \
|
||||
--cuda-graph-bs 2 4 6 8 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 \
|
||||
--moe-a2a-backend ascend_fuseep \
|
||||
--deepep-mode auto \
|
||||
--quantization modelslim \
|
||||
@@ -437,7 +437,7 @@ python3 -m sglang.launch_server \
|
||||
--prefill-max-requests 3 \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 8192 \
|
||||
--cuda-graph-bs 1 2 3 4 5 6 \
|
||||
--cuda-graph-bs-decode 1 2 3 4 5 6 \
|
||||
--moe-a2a-backend ascend_fuseep \
|
||||
--deepep-mode auto \
|
||||
--quantization modelslim \
|
||||
@@ -541,7 +541,7 @@ python3 -m sglang.launch_server \
|
||||
--enable-prefill-delayer \
|
||||
--chunked-prefill-size 196608 \
|
||||
--max-prefill-tokens 8192 \
|
||||
--cuda-graph-bs 1 2 4 8 12 16 20 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 12 16 20 \
|
||||
--moe-a2a-backend ascend_fuseep \
|
||||
--fuseep-mode 2 \
|
||||
--quantization modelslim \
|
||||
|
||||
+2
-2
@@ -101,7 +101,7 @@ python3 -m sglang.launch_server \
|
||||
--enable-dp-lm-head \
|
||||
--tp 16 \
|
||||
--mem-fraction-static 0.78 \
|
||||
--cuda-graph-bs 1 \
|
||||
--cuda-graph-bs-decode 1 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen25
|
||||
```
|
||||
@@ -210,7 +210,7 @@ python3 -m sglang.launch_server \
|
||||
--enable-dp-attention \
|
||||
--enable-dp-lm-head \
|
||||
--mem-fraction-static 0.8 \
|
||||
--cuda-graph-bs 1 2 4 8 16 20 24 26 27 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen25
|
||||
```
|
||||
|
||||
+4
-4
@@ -109,7 +109,7 @@ python3 -m sglang.launch_server \
|
||||
--enable-dp-attention \
|
||||
--dp-size 2 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--cuda-graph-bs 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen
|
||||
@@ -211,7 +211,7 @@ python3 -m sglang.launch_server \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.87 \
|
||||
--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
|
||||
--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen
|
||||
@@ -311,7 +311,7 @@ python3 -m sglang.launch_server \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.87 \
|
||||
--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
|
||||
--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen
|
||||
@@ -411,7 +411,7 @@ python3 -m sglang.launch_server \
|
||||
--max-prefill-tokens 35000 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \
|
||||
--cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen
|
||||
|
||||
+3
-3
@@ -100,7 +100,7 @@ python3 -m sglang.launch_server \
|
||||
--speculative-num-draft-tokens 5 \
|
||||
--tp-size 16 \
|
||||
--mem-fraction-static 0.72 \
|
||||
--cuda-graph-bs 1 \
|
||||
--cuda-graph-bs-decode 1 \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen
|
||||
@@ -199,7 +199,7 @@ python3 -m sglang.launch_server \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tp-size 4 \
|
||||
--mem-fraction-static 0.845 \
|
||||
--cuda-graph-bs 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
|
||||
--cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen
|
||||
@@ -298,7 +298,7 @@ python3 -m sglang.launch_server \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tp-size 4 \
|
||||
--mem-fraction-static 0.845 \
|
||||
--cuda-graph-bs 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
|
||||
--cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen
|
||||
|
||||
+9
-9
@@ -111,7 +111,7 @@ python3 -m sglang.launch_server \
|
||||
--trust-remote-code \
|
||||
--max-running-requests 16 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--cuda-graph-bs 2 3 4 5 6 8 10 12 14 16 \
|
||||
--cuda-graph-bs-decode 2 3 4 5 6 8 10 12 14 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--moe-a2a-backend deepep \
|
||||
@@ -330,7 +330,7 @@ python3 -m sglang.launch_server \
|
||||
--max-prefill-tokens 131072 \
|
||||
--max-mamba-cache-size 320 \
|
||||
--prefill-max-requests 10 \
|
||||
--mamba-scheduler-strategy extra_buffer \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--trust-remote-code \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.6 \
|
||||
@@ -451,7 +451,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 48 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-total-tokens 210000 \
|
||||
--cuda-graph-bs 2 4 6 8 10 12 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 10 12 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--moe-a2a-backend deepep \
|
||||
@@ -569,7 +569,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 144 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-total-tokens 635000 \
|
||||
--cuda-graph-bs 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--moe-a2a-backend deepep \
|
||||
@@ -686,7 +686,7 @@ python3 -m sglang.launch_server \
|
||||
--trust-remote-code \
|
||||
--max-running-requests 160 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--moe-a2a-backend deepep \
|
||||
@@ -803,7 +803,7 @@ python3 -m sglang.launch_server \
|
||||
--trust-remote-code \
|
||||
--max-running-requests 432 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--cuda-graph-bs 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--moe-a2a-backend deepep \
|
||||
@@ -921,7 +921,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 16 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--max-total-tokens 1065000 \
|
||||
--cuda-graph-bs 2 4 6 8 10 12 14 16 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--moe-a2a-backend deepep \
|
||||
@@ -1039,7 +1039,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 32 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--max-total-tokens 1065000 \
|
||||
--cuda-graph-bs 2 4 6 8 12 14 16 \
|
||||
--cuda-graph-bs-decode 2 4 6 8 12 14 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--moe-a2a-backend deepep \
|
||||
@@ -1147,7 +1147,7 @@ python3 -m sglang.launch_server \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 65536 \
|
||||
--max-mamba-cache-size 640 \
|
||||
--mamba-scheduler-strategy extra_buffer \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--trust-remote-code \
|
||||
--max-running-requests 128 \
|
||||
--mem-fraction-static 0.6 \
|
||||
|
||||
+9
-9
@@ -96,7 +96,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 60 \
|
||||
--max-mamba-cache-size 60 \
|
||||
--mem-fraction-static 0.74 \
|
||||
--cuda-graph-bs 2 4 8 14 16 24 26 32 36 37 40 42 44 45 46 50 52 60 \
|
||||
--cuda-graph-bs-decode 2 4 8 14 16 24 26 32 36 37 40 42 44 45 46 50 52 60 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -194,7 +194,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 38 \
|
||||
--max-mamba-cache-size 38 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--cuda-graph-bs 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \
|
||||
--cuda-graph-bs-decode 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \
|
||||
--enable-prefill-delayer \
|
||||
--prefill-delayer-queue-min-ratio 0.45 \
|
||||
--prefill-delayer-max-delay-ms 5500 \
|
||||
@@ -292,12 +292,12 @@ python3 -m sglang.launch_server \
|
||||
--device npu \
|
||||
--chunked-prefill-size 32768 \
|
||||
--max-prefill-tokens 32768 \
|
||||
--mamba-scheduler-strategy extra_buffer \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--trust-remote-code \
|
||||
--max-running-requests 20 \
|
||||
--max-mamba-cache-size 160 \
|
||||
--mem-fraction-static 0.82 \
|
||||
--cuda-graph-bs 1 2 5 10 15 17 19 20 \
|
||||
--cuda-graph-bs-decode 1 2 5 10 15 17 19 20 \
|
||||
--dtype bfloat16 \
|
||||
--mamba-ssm-dtype bfloat16 \
|
||||
--speculative-algorithm NEXTN \
|
||||
@@ -397,7 +397,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 64 \
|
||||
--max-mamba-cache-size 74 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--cuda-graph-bs 2 8 16 32 40 45 50 54 \
|
||||
--cuda-graph-bs-decode 2 8 16 32 40 45 50 54 \
|
||||
--enable-multimodal \
|
||||
--quantization modelslim \
|
||||
--mm-attention-backend ascend_attn \
|
||||
@@ -491,7 +491,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 6 \
|
||||
--max-mamba-cache-size 16 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--cuda-graph-bs 1 2 4 5 6 \
|
||||
--cuda-graph-bs-decode 1 2 4 5 6 \
|
||||
--quantization modelslim \
|
||||
--dtype bfloat16 \
|
||||
--mamba-ssm-dtype bfloat16 \
|
||||
@@ -587,7 +587,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 6 \
|
||||
--max-mamba-cache-size 7 \
|
||||
--mem-fraction-static 0.63 \
|
||||
--cuda-graph-bs 1 2 4 5 6 \
|
||||
--cuda-graph-bs-decode 1 2 4 5 6 \
|
||||
--enable-multimodal \
|
||||
--quantization modelslim \
|
||||
--mm-attention-backend ascend_attn \
|
||||
@@ -682,7 +682,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 37 \
|
||||
--max-mamba-cache-size 74 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--cuda-graph-bs 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \
|
||||
--cuda-graph-bs-decode 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \
|
||||
--quantization modelslim \
|
||||
--dtype bfloat16 \
|
||||
--mamba-ssm-dtype bfloat16 \
|
||||
@@ -779,7 +779,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 28 \
|
||||
--max-mamba-cache-size 50 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--cuda-graph-bs 2 4 6 \
|
||||
--cuda-graph-bs-decode 2 4 6 \
|
||||
--enable-multimodal \
|
||||
--quantization modelslim \
|
||||
--mm-attention-backend ascend_attn \
|
||||
|
||||
+11
-11
@@ -103,7 +103,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 120 \
|
||||
--max-mamba-cache-size 120 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--cuda-graph-bs 4 16 32 48 64 110 165 \
|
||||
--cuda-graph-bs-decode 4 16 32 48 64 110 165 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -205,7 +205,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 42 \
|
||||
--max-mamba-cache-size 42 \
|
||||
--mem-fraction-static 0.75 \
|
||||
--cuda-graph-bs 4 8 16 24 48 64 80 \
|
||||
--cuda-graph-bs-decode 4 8 16 24 48 64 80 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -308,7 +308,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 3 \
|
||||
--max-mamba-cache-size 3 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--cuda-graph-bs 1 2 3 \
|
||||
--cuda-graph-bs-decode 1 2 3 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -402,11 +402,11 @@ python3 -m sglang.launch_server \
|
||||
--max-prefill-tokens 65536 \
|
||||
--trust-remote-code \
|
||||
--enable-prefill-delayer \
|
||||
--mamba-scheduler-strategy extra_buffer \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--max-running-requests 103 \
|
||||
--max-mamba-cache-size 85 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--cuda-graph-bs 2 4 8 16 32 48 64 80 96 103 \
|
||||
--cuda-graph-bs-decode 2 4 8 16 32 48 64 80 96 103 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -507,7 +507,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 1 \
|
||||
--max-mamba-cache-size 6 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--cuda-graph-bs 1 \
|
||||
--cuda-graph-bs-decode 1 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -606,7 +606,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 122 \
|
||||
--max-mamba-cache-size 122 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--cuda-graph-bs 4 16 32 64 96 116 120 122 \
|
||||
--cuda-graph-bs-decode 4 16 32 64 96 116 120 122 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -705,7 +705,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 10 \
|
||||
--max-mamba-cache-size 20 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--cuda-graph-bs 2 4 8 12 14 16 \
|
||||
--cuda-graph-bs-decode 2 4 8 12 14 16 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -800,11 +800,11 @@ python3 -m sglang.launch_server \
|
||||
--max-total-tokens 470784 \
|
||||
--max-prefill-tokens 65536 \
|
||||
--trust-remote-code \
|
||||
--mamba-scheduler-strategy extra_buffer \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--max-running-requests 40 \
|
||||
--max-mamba-cache-size 200 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--cuda-graph-bs 2 8 16 24 32 36 40 \
|
||||
--cuda-graph-bs-decode 2 8 16 24 32 36 40 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
@@ -906,7 +906,7 @@ python3 -m sglang.launch_server \
|
||||
--max-running-requests 1 \
|
||||
--max-mamba-cache-size 6 \
|
||||
--mem-fraction-static 0.68 \
|
||||
--cuda-graph-bs 1 \
|
||||
--cuda-graph-bs-decode 1 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16 \
|
||||
|
||||
+3
-3
@@ -95,7 +95,7 @@ python3 -m sglang.launch_server \
|
||||
--chunked-prefill-size 16384 \
|
||||
--tp-size 1 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--cuda-graph-bs 8 12 24 36 48 51 55 60 63 64 66 68 70 \
|
||||
--cuda-graph-bs-decode 8 12 24 36 48 51 55 60 63 64 66 68 70 \
|
||||
--dtype bfloat16 \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
@@ -192,7 +192,7 @@ python3 -m sglang.launch_server \
|
||||
--chunked-prefill-size -1 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.894 \
|
||||
--cuda-graph-bs 1 \
|
||||
--cuda-graph-bs-decode 1 \
|
||||
--dtype bfloat16 \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
@@ -289,7 +289,7 @@ python3 -m sglang.launch_server \
|
||||
--chunked-prefill-size -1 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.894 \
|
||||
--cuda-graph-bs 1 5 15 16 \
|
||||
--cuda-graph-bs-decode 1 5 15 16 \
|
||||
--dtype bfloat16 \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
|
||||
+3
-3
@@ -116,7 +116,7 @@ python3 -m sglang.launch_server \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-running-requests 2 \
|
||||
--cuda-graph-bs 2 \
|
||||
--cuda-graph-bs-decode 2 \
|
||||
--mamba-ssm-dtype bfloat16 \
|
||||
--speculative-draft-model-path $DRAFT_MODEL_PATH \
|
||||
--reasoning-parser qwen3 \
|
||||
@@ -235,7 +235,7 @@ python3 -m sglang.launch_server \
|
||||
--enable-dp-lm-head \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--cuda-graph-bs 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \
|
||||
--cuda-graph-bs-decode 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
@@ -349,7 +349,7 @@ python3 -m sglang.launch_server \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-running-requests 16 \
|
||||
--cuda-graph-bs 2 4 8 \
|
||||
--cuda-graph-bs-decode 2 4 8 \
|
||||
--mamba-ssm-dtype bfloat16 \
|
||||
--speculative-draft-model-path $DRAFT_MODEL_PATH \
|
||||
--reasoning-parser qwen3 \
|
||||
|
||||
@@ -27,7 +27,7 @@ v0.5.16 or a later version.
|
||||
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
|
||||
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 20 21 22` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 20 21 22` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ recommended to use v0.5.16 or a later version.
|
||||
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6 7 8` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6 7 8` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
|
||||
+1
-1
@@ -27,7 +27,7 @@ recommended to use v0.5.16 or a later version.
|
||||
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
|
||||
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 24 36 40 48 56` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
@@ -31,7 +31,7 @@ v0.5.16 or a later version.
|
||||
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
@@ -23,7 +23,7 @@ multi-node deployment, prefill-decode disaggregation, feature configuration, and
|
||||
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 16` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
@@ -220,7 +220,7 @@ python3 -m sglang.launch_server \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-bs 16 \
|
||||
--cuda-graph-bs-decode 16 \
|
||||
--quantization modelslim \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
|
||||
@@ -302,7 +302,7 @@ do
|
||||
--mem-fraction-static 0.8 \
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-max-bs 32 \
|
||||
--cuda-graph-max-bs-decode 32 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
@@ -421,8 +421,8 @@ do
|
||||
--port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --ep-size 16 \
|
||||
--mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \
|
||||
--cuda-graph-max-bs 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
|
||||
--tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \
|
||||
--cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
|
||||
--tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \
|
||||
--speculative-draft-model-quantization unquant \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
|
||||
NODE_RANK=$i
|
||||
|
||||
@@ -21,7 +21,7 @@ multi-node (Atlas 800I A2) PD mixed mode and speculative decoding.
|
||||
| Feature | Example usage |
|
||||
|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| Tensor Parallelism | `--tp-size 16` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs`; e.g. `--cuda-graph-bs 4 8 16 20 24 28 32 36 40` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode`; e.g. `--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| Reasoning Mode | `extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}` — deep chain-of-thought;<br/>`"reasoning_effort": "low"` — short thinking chain;<br/>`"reasoning_effort": "no_think"` — direct response, no thinking |
|
||||
@@ -209,7 +209,7 @@ python3 -m sglang.launch_server \
|
||||
--base-gpu-id 0 \
|
||||
--prefill-max-requests 40 \
|
||||
--max-running-requests 40 \
|
||||
--cuda-graph-bs 4 8 16 20 24 28 32 36 40 \
|
||||
--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 2 \
|
||||
--speculative-eagle-topk 1 \
|
||||
@@ -286,7 +286,7 @@ for i in "${!NODE_IPS[@]}"; do
|
||||
--base-gpu-id 0 \
|
||||
--prefill-max-requests 40 \
|
||||
--max-running-requests 40 \
|
||||
--cuda-graph-bs 4 8 16 20 24 28 32 36 40 \
|
||||
--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 2 \
|
||||
--speculative-eagle-topk 1 \
|
||||
|
||||
@@ -30,7 +30,7 @@ v0.5.16 or a later version.
|
||||
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 24 32 48 64 96 120` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 4 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 5 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ v0.5.16 or a later version.
|
||||
| Expert Parallelism | `--moe-a2a-backend deepep \`<br/>`--deepep-mode low_latency` |
|
||||
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--enable-multi-layer-eagle` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ v0.5.16 or a later version.
|
||||
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend ascend_fuseep \`<br/>`--deepep-mode auto` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 8 16 24 32 48 64 80` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 8 16 24 32 48 64 80` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ v0.5.16 or a later version.
|
||||
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 94208` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ v0.5.16 or a later version.
|
||||
| Tensor Parallelism | `--tp-size 2` |
|
||||
| Data Parallelism | `--dp-size 2` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ version.
|
||||
|-------------------------------|-----------------------------------------------------------------------------------------------|
|
||||
| Tensor Parallelism | `--tp-size 4` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 4 8 16` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 4 8 16` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ use v0.5.16 or a later version.
|
||||
| Data Parallelism | `--dp-size 8` |
|
||||
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
@@ -26,7 +26,7 @@ v0.5.16 or a later version.
|
||||
| Tensor Parallelism | `--tp-size 2` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 8 16 32 48` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 8 16 32 48` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ version**.
|
||||
|-------------------------------|-----------------------------------------------------------------------------------------------|
|
||||
| Tensor Parallelism | `--tp-size 2` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 16 24 32 48 64 80 96 112 120` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
|
||||
|
||||
@@ -28,7 +28,7 @@ source.
|
||||
| Expert Parallelism | `--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 16` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
@@ -215,7 +215,7 @@ do
|
||||
--enable-dp-attention --dp-size 4 --enable-dp-lm-head \
|
||||
--mem-fraction-static 0.8 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--cuda-graph-bs 16 \
|
||||
--cuda-graph-bs-decode 16 \
|
||||
--disable-radix-cache \
|
||||
--max-running-requests 64 \
|
||||
--host 0.0.0.0 \
|
||||
|
||||
@@ -26,7 +26,7 @@ version.
|
||||
| Tensor Parallelism | `--tp-size 2` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 6 9 10 15 16` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 6 9 10 15 16` |
|
||||
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
|
||||
|
||||
+1
-1
@@ -27,7 +27,7 @@ v0.5.16 or a later version.
|
||||
| Data Parallelism | `--dp-size 2` |
|
||||
| Expert Parallelism | `--ep-size 4 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 8` |
|
||||
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 8` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
@@ -755,9 +755,9 @@ see [Quantization on Ascend](/docs/hardware-platforms/ascend-npus/optimization/q
|
||||
|
||||
Breaks large prefill computations into smaller, fixed-size chunks for better scheduling interleaving with decode batches. This prevents long prefill requests from blocking decode latency. Use `-1` to disable chunked prefill on dedicated prefill nodes. Has partial compatibility with most features because chunk boundaries introduce scheduling complexity.
|
||||
|
||||
### NPU Graph (`--cuda-graph-bs`)
|
||||
### NPU Graph (`--cuda-graph-bs-decode`)
|
||||
|
||||
Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes.
|
||||
Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs-decode`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes.
|
||||
|
||||
<Note>
|
||||
`--enable-torch-compile` is incompatible with NPU Graph. When torch.compile is enabled, NPU Graph must be disabled via `--disable-cuda-graph`.
|
||||
|
||||
@@ -816,12 +816,6 @@ click [Server Arguments](../../../advanced_features/server_arguments).
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>List[float]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--collect-tokens-histogram`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag<br/> (set to enable)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prompt-tokens-buckets`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
|
||||
@@ -1498,7 +1492,7 @@ non-default speculative acceptance thresholds or deterministic inference.
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--speculative-dflash-draft-window-size`<br/>`--speculative-draft-window-size`</td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--speculative-draft-window-size`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
@@ -2073,7 +2067,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-backend-prefill`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`disabled`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`disabled`, `tc_piecewise`<br/> (`tc_piecewise` currently supports Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -2113,7 +2107,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-bs`</td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-bs-decode`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>List[int]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
@@ -2257,19 +2251,13 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enforce-piecewise-cuda-graph`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag<br/> (set to enable); <br/> Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-tokens`</td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-bs-prefill`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: JSON<br/> list</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-compiler`</td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-tc-compiler`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`eager`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`eager`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
@@ -2281,7 +2269,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--piecewise-cuda-graph-max-tokens`</td>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cuda-graph-max-bs-prefill`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
|
||||
@@ -223,7 +223,7 @@ sglang serve \
|
||||
--model-path ./zai-org/GLM-Image/vision_language_encoder/ \
|
||||
--tokenizer-path ./zai-org/GLM-Image/processor/ \
|
||||
--enable-multimodal \
|
||||
--cuda-graph-max-bs 28 \
|
||||
--cuda-graph-max-bs-decode 28 \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--disable-fast-image-processor \
|
||||
|
||||
@@ -124,7 +124,7 @@ export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199"
|
||||
export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399"
|
||||
sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \
|
||||
--tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal \
|
||||
--cuda-graph-bs 1 --device npu --attention-backend ascend --image-processor-backend pil \
|
||||
--cuda-graph-bs-decode 1 --device npu --attention-backend ascend --image-processor-backend pil \
|
||||
--tp-size 4 --port ${PORT} --mem-fraction-static 0.4
|
||||
```
|
||||
Second terminal with diffusion server:
|
||||
|
||||
Reference in New Issue
Block a user