minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU] Modify the parameter name and optional values, and add the parameter restrictions. Modify some parameters supported type. (#22804)

Browse Source
This commit is contained in:
chx96642264
2026-04-14 21:34:07 +08:00
committed by GitHub
parent 1588856e9b
commit 680bd4b429
1 changed files with 83 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/platforms/ascend/ascend_npu_support_features.md
+83 -83
View File
@@ -69,8 +69,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
| `--priority-scheduling-`<br/>`preemption-threshold` | `10` | Type: int | A2, A3 |
| `--schedule-conservativeness` | `1.0` | Type: float | A2, A3 |
| `--page-size` | `128` | Type: int | A2, A3 |
| `--swa-full-tokens-ratio` | `0.8` | Type: float | A2, A3 |
| `--disable-hybrid-swa-memory` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--swa-full-tokens-ratio` | `0.8` | Type: float | Planned |
| `--disable-hybrid-swa-memory` | `False` | bool flag<br/> (set to enable) | Planned |
| `--radix-eviction-policy` | `lru` | `lru`,<br/>`lfu` | A2, A3 |
| `--enable-prefill-delayer` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--prefill-delayer-max-delay-passes` | `30` | Type: int | A2, A3 |
@@ -78,32 +78,32 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
| `--prefill-delayer-forward-passes-buckets` | `None` | List[float] | A2, A3 |
| `--prefill-delayer-wait-seconds-buckets` | `None` | List[float] | A2, A3 |
| `--abort-on-priority-`<br/>`when-disabled` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-dynamic-chunking` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-dynamic-chunking` | `False` | bool flag<br/> (set to enable) | Experimental |
## Runtime options
| Argument | Defaults | Options | Server supported |
|----------------------------------------------------|----------|---------------------------|:----------------:|
| `--device` | `None` | Type: str | A2, A3 |
| `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | A2, A3 |
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int; Currently `2` not supported | A2, A3 |
| `--attention-context-parallel-size`<br/>`--attn-cp-size` | `1` | Type: int; must be equal to --tp-size | A2, A3 |
| `--moe-data-parallel-size`<br/>`--moe-dp-size` | `1` | Type: int | A2, A3 |
| `--pp-max-micro-batch-size` | `None` | Type: int | A2, A3 |
| `--pp-async-batch-depth` | `None` | Type: int | A2, A3 |
| `--stream-interval` | `1` | Type: int | A2, A3 |
| `--incremental-streaming-output` | `False` | bool flag (set to enable) | A2, A3 |
| `--random-seed` | `None` | Type: int | A2, A3 |
| `--constrained-json-`<br/>`whitespace-pattern` | `None` | Type: str | A2, A3 |
| `--constrained-json-`<br/>`disable-any-whitespace` | `False` | bool flag (set to enable) | A2, A3 |
| `--watchdog-timeout` | `300` | Type: float | A2, A3 |
| `--soft-watchdog-timeout` | `300` | Type: float | A2, A3 |
| `--dist-timeout` | `None` | Type: int | A2, A3 |
| `--download-dir` | `None` | Type: str | A2, A3 |
| `--model-checksum` | `None` | Type: str | A2, A3 |
| `--base-gpu-id` | `0` | Type: int | A2, A3 |
| `--gpu-id-step` | `1` | Type: int | A2, A3 |
| `--sleep-on-idle` | `False` | bool flag (set to enable) | A2, A3 |
| Argument | Defaults | Options | Server supported |
|----------------------------------------------------------|----------|----------------------------------------|:----------------:|
| `--device` | `None` | Type: str | A2, A3 |
| `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | A2, A3 |
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int; Currently `2` not supported | Experimental |
| `--attention-context-parallel-size`<br/>`--attn-cp-size` | `1` | Type: int; must be equal to --tp-size | A2, A3 |
| `--moe-data-parallel-size`<br/>`--moe-dp-size` | `1` | Type: int | Planned |
| `--pp-max-micro-batch-size` | `None` | Type: int | Experimental |
| `--pp-async-batch-depth` | `None` | Type: int | Experimental |
| `--stream-interval` | `1` | Type: int | A2, A3 |
| `--incremental-streaming-output` | `False` | bool flag (set to enable) | A2, A3 |
| `--random-seed` | `None` | Type: int | A2, A3 |
| `--constrained-json-`<br/>`whitespace-pattern` | `None` | Type: str | A2, A3 |
| `--constrained-json-`<br/>`disable-any-whitespace` | `False` | bool flag (set to enable) | A2, A3 |
| `--watchdog-timeout` | `300` | Type: float | A2, A3 |
| `--soft-watchdog-timeout` | `300` | Type: float | A2, A3 |
| `--dist-timeout` | `None` | Type: int | A2, A3 |
| `--download-dir` | `None` | Type: str | A2, A3 |
| `--model-checksum` | `None` | Type: str | Planned |
| `--base-gpu-id` | `0` | Type: int | A2, A3 |
| `--gpu-id-step` | `1` | Type: int | A2, A3 |
| `--sleep-on-idle` | `False` | bool flag (set to enable) | A2, A3 |
## Logging
@@ -315,64 +315,64 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
## Optimization/debug options
| Argument | Defaults | Options | Server supported |
|---------------------------------------------------------|----------|--------------------------------|:----------------:|
| `--disable-radix-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--cuda-graph-max-bs` | `None` | Type: int | A2, A3 |
| `--cuda-graph-bs` | `None` | List[int] | A2, A3 |
| `--disable-cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-cuda-graph-`<br/>`padding` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-profile-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-cudagraph-gc` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-nccl-nvls` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-symm-mem` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--disable-flashinfer-`<br/>`cutlass-moe-fp4-allgather` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-tokenizer-`<br/>`batch-decode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-custom-`<br/>`all-reduce` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-mscclpp` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-torch-`<br/>`symm-mem` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--disable-overlap`<br/>`-schedule` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-mixed-`<br/>`chunk` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-dp-attention` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-dp-lm-head` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-two-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | Planned |
| `--enable-single-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--tbo-token-`<br/>`distribution-threshold` | `0.48` | Type: float | Planned |
| `--enable-torch-`<br/>`compile` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-torch-`<br/>`compile-debug-mode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-piecewise-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--piecewise-cuda-`<br/>`graph-tokens` | `None` | Type: JSON<br/> list | A2, A3 |
| `--piecewise-cuda-`<br/>`graph-compiler` | `eager` | ["eager", "inductor"] | A2, A3 |
| `--torch-compile-max-bs` | `32` | Type: int | A2, A3 |
| `--piecewise-cuda-`<br/>`graph-max-tokens` | `None` | Type: int | A2, A3 |
| `--torchao-config` | `` | Type: str | Special for GPU |
| `--enable-nan-detection` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-p2p-check` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--triton-attention-`<br/>`reduce-in-fp32` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--triton-attention-`<br/>`num-kv-splits` | `8` | Type: int | Special for GPU |
| `--triton-attention-`<br/>`split-tile-size` | `None` | Type: int | Special for GPU |
| `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-memory-saver` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-draft-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--disable-shared-`<br/>`experts-fusion` | `True` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-chunked-`<br/>`prefix-cache` | `True` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-return-`<br/>`routed-experts` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--scheduler-recv-`<br/>`interval` | `1` | Type: int | A2, A3 |
| `--numa-node` | `None` | List[int] | A2, A3 |
| `--enable-deterministic-`<br/>`inference` | `False` | bool flag<br/> (set to enable) | Planned |
| `--rl-on-policy-target` | `None` | `fsdp` | Planned |
| `--enable-layerwise-`<br/>`nvtx-marker` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-attn-tp-`<br/>`input-scattered` | `False` | bool flag<br/> (set to enable) | Experimental |
| `--enable-nsa-prefill-`<br/>`context-parallel` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-fused-qk-`<br/>`norm-rope` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| Argument | Defaults | Options | Server supported |
|---------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------|:----------------:|
| `--disable-radix-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--cuda-graph-max-bs` | `None` | Type: int | A2, A3 |
| `--cuda-graph-bs` | `None` | List[int] | A2, A3 |
| `--disable-cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-cuda-graph-`<br/>`padding` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-profile-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-cudagraph-gc` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-nccl-nvls` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-symm-mem` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--disable-flashinfer-`<br/>`cutlass-moe-fp4-allgather` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-tokenizer-`<br/>`batch-decode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-custom-`<br/>`all-reduce` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-mscclpp` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-torch-`<br/>`symm-mem` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--disable-overlap`<br/>`-schedule` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-mixed-`<br/>`chunk` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-dp-attention` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-dp-lm-head` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-two-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | Planned |
| `--enable-single-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--tbo-token-`<br/>`distribution-threshold` | `0.48` | Type: float | Planned |
| `--enable-torch-`<br/>`compile` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-torch-`<br/>`compile-debug-mode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enforce-piecewise-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable); <br/> Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported. | A2, A3 |
| `--piecewise-cuda-`<br/>`graph-tokens` | `None` | Type: JSON<br/> list | A2, A3 |
| `--piecewise-cuda-`<br/>`graph-compiler` | `eager` | `eager` | A2, A3 |
| `--torch-compile-max-bs` | `32` | Type: int | A2, A3 |
| `--piecewise-cuda-`<br/>`graph-max-tokens` | `None` | Type: int | A2, A3 |
| `--torchao-config` | `` | Type: str | Special for GPU |
| `--enable-nan-detection` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-p2p-check` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--triton-attention-`<br/>`reduce-in-fp32` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--triton-attention-`<br/>`num-kv-splits` | `8` | Type: int | Special for GPU |
| `--triton-attention-`<br/>`split-tile-size` | `None` | Type: int | Special for GPU |
| `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-memory-saver` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-draft-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--disable-shared-`<br/>`experts-fusion` | `True` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-chunked-`<br/>`prefix-cache` | `True` | bool flag<br/> (set to enable) | A2, A3 |
| `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-return-`<br/>`routed-experts` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--scheduler-recv-`<br/>`interval` | `1` | Type: int | A2, A3 |
| `--numa-node` | `None` | List[int] | A2, A3 |
| `--enable-deterministic-`<br/>`inference` | `False` | bool flag<br/> (set to enable) | Planned |
| `--rl-on-policy-target` | `None` | `fsdp` | Planned |
| `--enable-layerwise-`<br/>`nvtx-marker` | `False` | bool flag<br/> (set to enable) | Special for GPU |
| `--enable-attn-tp-`<br/>`input-scattered` | `False` | bool flag<br/> (set to enable) | Experimental |
| `--enable-nsa-prefill-`<br/>`context-parallel` | `False` | bool flag<br/> (set to enable) | A2, A3 |
| `--enable-fused-qk-`<br/>`norm-rope` | `False` | bool flag<br/> (set to enable) | Special for GPU |
## Dynamic batch tokenizer