diff --git a/docs/platforms/ascend/ascend_npu_support_features.md b/docs/platforms/ascend/ascend_npu_support_features.md index dba352de8..83f955ed6 100644 --- a/docs/platforms/ascend/ascend_npu_support_features.md +++ b/docs/platforms/ascend/ascend_npu_support_features.md @@ -69,8 +69,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--priority-scheduling-`
`preemption-threshold` | `10` | Type: int | A2, A3 | | `--schedule-conservativeness` | `1.0` | Type: float | A2, A3 | | `--page-size` | `128` | Type: int | A2, A3 | -| `--swa-full-tokens-ratio` | `0.8` | Type: float | A2, A3 | -| `--disable-hybrid-swa-memory` | `False` | bool flag
(set to enable) | A2, A3 | +| `--swa-full-tokens-ratio` | `0.8` | Type: float | Planned | +| `--disable-hybrid-swa-memory` | `False` | bool flag
(set to enable) | Planned | | `--radix-eviction-policy` | `lru` | `lru`,
`lfu` | A2, A3 | | `--enable-prefill-delayer` | `False` | bool flag
(set to enable) | A2, A3 | | `--prefill-delayer-max-delay-passes` | `30` | Type: int | A2, A3 | @@ -78,32 +78,32 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--prefill-delayer-forward-passes-buckets` | `None` | List[float] | A2, A3 | | `--prefill-delayer-wait-seconds-buckets` | `None` | List[float] | A2, A3 | | `--abort-on-priority-`
`when-disabled` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-dynamic-chunking` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-dynamic-chunking` | `False` | bool flag
(set to enable) | Experimental | ## Runtime options -| Argument | Defaults | Options | Server supported | -|----------------------------------------------------|----------|---------------------------|:----------------:| -| `--device` | `None` | Type: str | A2, A3 | -| `--tensor-parallel-size`
`--tp-size` | `1` | Type: int | A2, A3 | -| `--pipeline-parallel-size`
`--pp-size` | `1` | Type: int; Currently `2` not supported | A2, A3 | -| `--attention-context-parallel-size`
`--attn-cp-size` | `1` | Type: int; must be equal to --tp-size | A2, A3 | -| `--moe-data-parallel-size`
`--moe-dp-size` | `1` | Type: int | A2, A3 | -| `--pp-max-micro-batch-size` | `None` | Type: int | A2, A3 | -| `--pp-async-batch-depth` | `None` | Type: int | A2, A3 | -| `--stream-interval` | `1` | Type: int | A2, A3 | -| `--incremental-streaming-output` | `False` | bool flag (set to enable) | A2, A3 | -| `--random-seed` | `None` | Type: int | A2, A3 | -| `--constrained-json-`
`whitespace-pattern` | `None` | Type: str | A2, A3 | -| `--constrained-json-`
`disable-any-whitespace` | `False` | bool flag (set to enable) | A2, A3 | -| `--watchdog-timeout` | `300` | Type: float | A2, A3 | -| `--soft-watchdog-timeout` | `300` | Type: float | A2, A3 | -| `--dist-timeout` | `None` | Type: int | A2, A3 | -| `--download-dir` | `None` | Type: str | A2, A3 | -| `--model-checksum` | `None` | Type: str | A2, A3 | -| `--base-gpu-id` | `0` | Type: int | A2, A3 | -| `--gpu-id-step` | `1` | Type: int | A2, A3 | -| `--sleep-on-idle` | `False` | bool flag (set to enable) | A2, A3 | +| Argument | Defaults | Options | Server supported | +|----------------------------------------------------------|----------|----------------------------------------|:----------------:| +| `--device` | `None` | Type: str | A2, A3 | +| `--tensor-parallel-size`
`--tp-size` | `1` | Type: int | A2, A3 | +| `--pipeline-parallel-size`
`--pp-size` | `1` | Type: int; Currently `2` not supported | Experimental | +| `--attention-context-parallel-size`
`--attn-cp-size` | `1` | Type: int; must be equal to --tp-size | A2, A3 | +| `--moe-data-parallel-size`
`--moe-dp-size` | `1` | Type: int | Planned | +| `--pp-max-micro-batch-size` | `None` | Type: int | Experimental | +| `--pp-async-batch-depth` | `None` | Type: int | Experimental | +| `--stream-interval` | `1` | Type: int | A2, A3 | +| `--incremental-streaming-output` | `False` | bool flag (set to enable) | A2, A3 | +| `--random-seed` | `None` | Type: int | A2, A3 | +| `--constrained-json-`
`whitespace-pattern` | `None` | Type: str | A2, A3 | +| `--constrained-json-`
`disable-any-whitespace` | `False` | bool flag (set to enable) | A2, A3 | +| `--watchdog-timeout` | `300` | Type: float | A2, A3 | +| `--soft-watchdog-timeout` | `300` | Type: float | A2, A3 | +| `--dist-timeout` | `None` | Type: int | A2, A3 | +| `--download-dir` | `None` | Type: str | A2, A3 | +| `--model-checksum` | `None` | Type: str | Planned | +| `--base-gpu-id` | `0` | Type: int | A2, A3 | +| `--gpu-id-step` | `1` | Type: int | A2, A3 | +| `--sleep-on-idle` | `False` | bool flag (set to enable) | A2, A3 | ## Logging @@ -315,64 +315,64 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen ## Optimization/debug options -| Argument | Defaults | Options | Server supported | -|---------------------------------------------------------|----------|--------------------------------|:----------------:| -| `--disable-radix-cache` | `False` | bool flag
(set to enable) | A2, A3 | -| `--cuda-graph-max-bs` | `None` | Type: int | A2, A3 | -| `--cuda-graph-bs` | `None` | List[int] | A2, A3 | -| `--disable-cuda-graph` | `False` | bool flag
(set to enable) | A2, A3 | -| `--disable-cuda-graph-`
`padding` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-profile-`
`cuda-graph` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-cudagraph-gc` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-nccl-nvls` | `False` | bool flag
(set to enable) | Special for GPU | -| `--enable-symm-mem` | `False` | bool flag
(set to enable) | Special for GPU | -| `--disable-flashinfer-`
`cutlass-moe-fp4-allgather` | `False` | bool flag
(set to enable) | Special for GPU | -| `--enable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | A2, A3 | -| `--disable-tokenizer-`
`batch-decode` | `False` | bool flag
(set to enable) | A2, A3 | -| `--disable-custom-`
`all-reduce` | `False` | bool flag
(set to enable) | Special for GPU | -| `--enable-mscclpp` | `False` | bool flag
(set to enable) | Special for GPU | -| `--enable-torch-`
`symm-mem` | `False` | bool flag
(set to enable) | Special for GPU | -| `--disable-overlap`
`-schedule` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-mixed-`
`chunk` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-dp-attention` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-dp-lm-head` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-two-`
`batch-overlap` | `False` | bool flag
(set to enable) | Planned | -| `--enable-single-`
`batch-overlap` | `False` | bool flag
(set to enable) | A2, A3 | -| `--tbo-token-`
`distribution-threshold` | `0.48` | Type: float | Planned | -| `--enable-torch-`
`compile` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-torch-`
`compile-debug-mode` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-piecewise-`
`cuda-graph` | `False` | bool flag
(set to enable) | A2, A3 | -| `--piecewise-cuda-`
`graph-tokens` | `None` | Type: JSON
list | A2, A3 | -| `--piecewise-cuda-`
`graph-compiler` | `eager` | ["eager", "inductor"] | A2, A3 | -| `--torch-compile-max-bs` | `32` | Type: int | A2, A3 | -| `--piecewise-cuda-`
`graph-max-tokens` | `None` | Type: int | A2, A3 | -| `--torchao-config` | `` | Type: str | Special for GPU | -| `--enable-nan-detection` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-p2p-check` | `False` | bool flag
(set to enable) | Special for GPU | -| `--triton-attention-`
`reduce-in-fp32` | `False` | bool flag
(set to enable) | Special for GPU | -| `--triton-attention-`
`num-kv-splits` | `8` | Type: int | Special for GPU | -| `--triton-attention-`
`split-tile-size` | `None` | Type: int | Special for GPU | -| `--delete-ckpt-`
`after-loading` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-memory-saver` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-draft-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | A2, A3 | -| `--allow-auto-truncate` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-custom-`
`logit-processor` | `False` | bool flag
(set to enable) | A2, A3 | -| `--flashinfer-mla-`
`disable-ragged` | `False` | bool flag
(set to enable) | Special for GPU | -| `--disable-shared-`
`experts-fusion` | `True` | bool flag
(set to enable) | A2, A3 | -| `--disable-chunked-`
`prefix-cache` | `True` | bool flag
(set to enable) | A2, A3 | -| `--disable-fast-`
`image-processor` | `False` | bool flag
(set to enable) | A2, A3 | -| `--keep-mm-feature-`
`on-device` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-return-`
`hidden-states` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-return-`
`routed-experts` | `False` | bool flag
(set to enable) | A2, A3 | -| `--scheduler-recv-`
`interval` | `1` | Type: int | A2, A3 | -| `--numa-node` | `None` | List[int] | A2, A3 | -| `--enable-deterministic-`
`inference` | `False` | bool flag
(set to enable) | Planned | -| `--rl-on-policy-target` | `None` | `fsdp` | Planned | -| `--enable-layerwise-`
`nvtx-marker` | `False` | bool flag
(set to enable) | Special for GPU | -| `--enable-attn-tp-`
`input-scattered` | `False` | bool flag
(set to enable) | Experimental | -| `--enable-nsa-prefill-`
`context-parallel` | `False` | bool flag
(set to enable) | A2, A3 | -| `--enable-fused-qk-`
`norm-rope` | `False` | bool flag
(set to enable) | Special for GPU | +| Argument | Defaults | Options | Server supported | +|---------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------|:----------------:| +| `--disable-radix-cache` | `False` | bool flag
(set to enable) | A2, A3 | +| `--cuda-graph-max-bs` | `None` | Type: int | A2, A3 | +| `--cuda-graph-bs` | `None` | List[int] | A2, A3 | +| `--disable-cuda-graph` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-cuda-graph-`
`padding` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-profile-`
`cuda-graph` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-cudagraph-gc` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-nccl-nvls` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-symm-mem` | `False` | bool flag
(set to enable) | Special for GPU | +| `--disable-flashinfer-`
`cutlass-moe-fp4-allgather` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-tokenizer-`
`batch-decode` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-custom-`
`all-reduce` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-mscclpp` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-torch-`
`symm-mem` | `False` | bool flag
(set to enable) | Special for GPU | +| `--disable-overlap`
`-schedule` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-mixed-`
`chunk` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-dp-attention` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-dp-lm-head` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-two-`
`batch-overlap` | `False` | bool flag
(set to enable) | Planned | +| `--enable-single-`
`batch-overlap` | `False` | bool flag
(set to enable) | A2, A3 | +| `--tbo-token-`
`distribution-threshold` | `0.48` | Type: float | Planned | +| `--enable-torch-`
`compile` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-torch-`
`compile-debug-mode` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enforce-piecewise-`
`cuda-graph` | `False` | bool flag
(set to enable);
Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported. | A2, A3 | +| `--piecewise-cuda-`
`graph-tokens` | `None` | Type: JSON
list | A2, A3 | +| `--piecewise-cuda-`
`graph-compiler` | `eager` | `eager` | A2, A3 | +| `--torch-compile-max-bs` | `32` | Type: int | A2, A3 | +| `--piecewise-cuda-`
`graph-max-tokens` | `None` | Type: int | A2, A3 | +| `--torchao-config` | `` | Type: str | Special for GPU | +| `--enable-nan-detection` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-p2p-check` | `False` | bool flag
(set to enable) | Special for GPU | +| `--triton-attention-`
`reduce-in-fp32` | `False` | bool flag
(set to enable) | Special for GPU | +| `--triton-attention-`
`num-kv-splits` | `8` | Type: int | Special for GPU | +| `--triton-attention-`
`split-tile-size` | `None` | Type: int | Special for GPU | +| `--delete-ckpt-`
`after-loading` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-memory-saver` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-draft-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | A2, A3 | +| `--allow-auto-truncate` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-custom-`
`logit-processor` | `False` | bool flag
(set to enable) | A2, A3 | +| `--flashinfer-mla-`
`disable-ragged` | `False` | bool flag
(set to enable) | Special for GPU | +| `--disable-shared-`
`experts-fusion` | `True` | bool flag
(set to enable) | A2, A3 | +| `--disable-chunked-`
`prefix-cache` | `True` | bool flag
(set to enable) | A2, A3 | +| `--disable-fast-`
`image-processor` | `False` | bool flag
(set to enable) | A2, A3 | +| `--keep-mm-feature-`
`on-device` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-return-`
`hidden-states` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-return-`
`routed-experts` | `False` | bool flag
(set to enable) | A2, A3 | +| `--scheduler-recv-`
`interval` | `1` | Type: int | A2, A3 | +| `--numa-node` | `None` | List[int] | A2, A3 | +| `--enable-deterministic-`
`inference` | `False` | bool flag
(set to enable) | Planned | +| `--rl-on-policy-target` | `None` | `fsdp` | Planned | +| `--enable-layerwise-`
`nvtx-marker` | `False` | bool flag
(set to enable) | Special for GPU | +| `--enable-attn-tp-`
`input-scattered` | `False` | bool flag
(set to enable) | Experimental | +| `--enable-nsa-prefill-`
`context-parallel` | `False` | bool flag
(set to enable) | A2, A3 | +| `--enable-fused-qk-`
`norm-rope` | `False` | bool flag
(set to enable) | Special for GPU | ## Dynamic batch tokenizer