minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[npu] update model and feature supported for ascend npu (#16390)

Browse Source
This commit is contained in:
Hexq0210
2026-01-04 21:50:20 +08:00
committed by GitHub
parent 27a0830593
commit bf32cd8397
2 changed files with 161 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/platforms/ascend_npu_support_features.md
+114 -27
View File
@@ -3,12 +3,13 @@
This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any
questions, please [open an issue](https://github.com/sgl-project/sglang/issues). questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
If you want to know the meaning and usage of each parameter, click [Service Arguments](https://docs.sglang.io/advanced_features/server_arguments.html). If you want to know the meaning and usage of each parameter,
click [Service Arguments](https://docs.sglang.io/advanced_features/server_arguments.html).
## Model and tokenizer ## Model and tokenizer
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
|----------------------------------------|----------|---------------------------------------|------------------------------------------|------------------------------------------| |----------------------------------------|----------|---------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--model-path`<br/>`--model` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--model-path`<br/>`--model` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-path` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--tokenizer-path` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-mode` | `auto` | `auto`, `slow` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--tokenizer-mode` | `auto` | `auto`, `slow` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -21,26 +22,36 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--is-embedding` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--is-embedding` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-multimodal` | `None` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--enable-multimodal` | `None` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--revision` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--revision` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--model-impl` | `None` | `auto`, `sglang`,<br/> `transformers` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | `--model-impl` | `None` | `auto`, `sglang`,<br/> `transformers` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## HTTP server ## HTTP server
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
|------------------------|-------------|--------------------------------|:----------------------------------------:|:----------------------------------------:| |-------------------------------------------------------------------|-------------|---------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--host` | `127.0.0.1` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--host` | `127.0.0.1` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--port` | `30000` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--port` | `30000` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--skip-server-warmup` | `False` | bool flag <br/>(set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--skip-server-warmup` | `False` | bool flag <br/>(set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--warmups` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--warmups` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--nccl-port` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--nccl-port` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--fastapi-root-path` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--grpc-mode` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--checkpoint-engine-` <br/> `wait-weights-` <br/> `before-ready` | FALSE | bool flag <br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Quantization and data type ## Quantization and data type
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
|-----------------------------|----------|-----------------------------------------|:----------------------------------------:|:----------------------------------------:| |-------------------------------------------------|----------|-----------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--dtype` | `auto` | `auto`,<br/> `float16`,<br/> `bfloat16` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--dtype` | `auto` | `auto`,<br/> `float16`,<br/> `bfloat16` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--quantization` | `None` | `modelslim` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--quantization` | `None` | `modelslim` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--quantization-param-path` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--quantization-param-path` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--kv-cache-dtype` | `auto` | `auto` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--kv-cache-dtype` | `auto` | `auto` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-fp32-lm-head` | FALSE | bool flag <br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--modelopt-quant` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--modelopt-checkpoint-`<br/>`restore-path` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--modelopt-checkpoint-`<br/>`save-path` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--modelopt-export-path` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--quantize-and-serve--`<br/>`rl-quant-profile` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--load-format flash_rl` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Memory and scheduling ## Memory and scheduling
@@ -62,6 +73,8 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--hybrid-kvcache-ratio` | `None` | Optional[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--hybrid-kvcache-ratio` | `None` | Optional[float] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--swa-full-tokens-ratio` | `0.8` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--swa-full-tokens-ratio` | `0.8` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-hybrid-swa-memory` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--disable-hybrid-swa-memory` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--abort-on-priority-`<br/>`when-disabled` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-dynamic-chunking` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Runtime options ## Runtime options
@@ -71,6 +84,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--pp-max-micro-batch-size` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--pp-max-micro-batch-size` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--pp-async-batch-depth` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--stream-interval` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--stream-interval` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--stream-output` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--stream-output` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--random-seed` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--random-seed` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -78,11 +92,11 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--constrained-json-`<br/>`disable-any-whitespace` | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--constrained-json-`<br/>`disable-any-whitespace` | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--watchdog-timeout` | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--watchdog-timeout` | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--soft-watchdog-timeout` | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--soft-watchdog-timeout` | `300` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dist-timeout` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--dist-timeout` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--base-gpu-id` | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--base-gpu-id` | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--gpu-id-step` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--gpu-id-step` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--sleep-on-idle` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--sleep-on-idle` | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--mm-process-config` | `{}` | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--custom-sigquit-handler` | `{}` | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Logging ## Logging
@@ -92,6 +106,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--log-level-http` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--log-level-http` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--log-requests` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--log-requests` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--log-requests-format` | text | text, json | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--crash-dump-folder` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--crash-dump-folder` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--crash-on-nan` | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--crash-on-nan` | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-metrics` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-metrics` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
@@ -106,11 +121,18 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--generation-tokens-buckets` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--generation-tokens-buckets` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--gc-warning-threshold-secs` | `0.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--gc-warning-threshold-secs` | `0.0` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--decode-log-interval` | `40` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--decode-log-interval` | `40` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-request-time-`<br/>`stats-logging` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-request-time-`<br/>`stats-logging` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--kv-events-config` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--kv-events-config` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-trace` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-trace` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--oltp-traces-endpoint` | `localhost:4317` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--oltp-traces-endpoint` | `localhost:4317` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## RequestMetricsExporter configuration
| Argument | Defaults | Options | A2 | A3 |
|---------------------------------------|----------|---------|:--------------------------------------:|:--------------------------------------:|
| `--export-metrics-to-`<br/>`file` | FALSE | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--export-metrics-to-`<br/>`file-dir` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## API related ## API related
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
@@ -132,6 +154,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|----------------------------------------|---------------|-------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:| |----------------------------------------|---------------|-------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--data-parallel-size`<br/>`--dp-size` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--data-parallel-size`<br/>`--dp-size` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--load-balance-method` | `round_robin` | `round_robin`,<br/> `shortest_queue`,<br/> `minimum_tokens` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--load-balance-method` | `round_robin` | `round_robin`,<br/> `shortest_queue`,<br/> `minimum_tokens` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--prefill-round-robin-balance` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Multi-node distributed serving ## Multi-node distributed serving
@@ -154,12 +177,12 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|--------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:| |--------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--enable-lora` | `False` | Bool flag <br/>(set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--enable-lora` | `False` | Bool flag <br/>(set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-lora-rank` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--max-lora-rank` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--lora-target-modules` | `None` | `q_proj`, <br/>`k_proj`,<br/> `v_proj`,<br/> `o_proj`,<br/> `gate_proj`,<br/> `up_proj`,<br/> `down_proj`,<br/> `qkv_proj`,<br/> `gate_up_proj`,<br/> `all` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--lora-target-modules` | `None` | `q_proj`, <br/>`k_proj`,<br/> `v_proj`,<br/> `o_proj`,<br/> `gate_proj`,<br/> `up_proj`,<br/> `down_proj`,<br/> `qkv_proj`,<br/> `gate_up_proj`,<br/> `all` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--lora-paths` | `None` | Type: List[str] /<br/> JSON objects | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--lora-paths` | `None` | Type: List[str] /<br/> JSON objects | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-loras-per-batch` | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--max-loras-per-batch` | `8` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-loaded-loras` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--max-loaded-loras` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--lora-eviction-policy` | `lru` | `lru`,<br/> `fifo` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--lora-eviction-policy` | `lru` | `lru`,<br/> `fifo` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--lora-backend` | `triton` | `triton`,<br/> `csgmv` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--lora-backend` | `triton` | `triton`,<br/> `csgmv` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--max-lora-chunk-size` | `16` | `16`, `32`,<br/> `64`, `128` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--max-lora-chunk-size` | `16` | `16`, `32`,<br/> `64`, `128` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Kernel Backends (Attention, Sampling, Grammar, GEMM) ## Kernel Backends (Attention, Sampling, Grammar, GEMM)
@@ -184,6 +207,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--speculative-algorithm` | `None` | `EAGLE`,<br/> `EAGLE3`,<br/> `NEXTN`,<br/> `STANDALONE`, <br/>`NGRAM` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--speculative-algorithm` | `None` | `EAGLE`,<br/> `EAGLE3`,<br/> `NEXTN`,<br/> `STANDALONE`, <br/>`NGRAM` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-model-path`<br/>`--speculative-draft-model` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--speculative-draft-model-path`<br/>`--speculative-draft-model` | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-model-`<br/>`revision` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--speculative-draft-model-`<br/>`revision` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-draft-load-format` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-num-steps` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--speculative-num-steps` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-eagle-topk` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--speculative-eagle-topk` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-num-draft-tokens` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--speculative-num-draft-tokens` | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -193,6 +217,8 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--speculative-attention-`<br/>`mode` | `prefill` | `prefill`,<br/> `decode` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--speculative-attention-`<br/>`mode` | `prefill` | `prefill`,<br/> `decode` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-moe-runner-`<br/>`backend` | `None` | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--speculative-moe-runner-`<br/>`backend` | `None` | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-moe-a2a-`<br/>`backend` | `None` | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--speculative-moe-a2a-`<br/>`backend` | `None` | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-attention-backend` | | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-model-quantization` | | | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Ngram speculative decoding ## Ngram speculative decoding
@@ -241,12 +267,6 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--mamba-scheduler-strategy` | `auto` | `auto`, <br/>`no_buffer`,<br/> `extra_buffer` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--mamba-scheduler-strategy` | `auto` | `auto`, <br/>`no_buffer`,<br/> `extra_buffer` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-track-interval` | `256` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--mamba-track-interval` | `256` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Args for multi-item scoring
| Argument | Defaults | Options | A2 | A3 |
|----------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
| `--multi-item-scoring-delimiter` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Hierarchical cache ## Hierarchical cache
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
@@ -268,6 +288,17 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
|--------------------|----------|--------------------------------|:--------------------------------------:|:--------------------------------------:| |--------------------|----------|--------------------------------|:--------------------------------------:|:--------------------------------------:|
| `--enable-lmcache` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-lmcache` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Ktransformer server args
| Argument | Defaults | Options | A2 | A3 |
|----------------------------------------------|----------|---------|:--------------------------------------:|:--------------------------------------:|
| `--kt-weight-path` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--kt-method` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--kt-cpuinfer` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--kt-threadpool-count` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--kt-num-gpu-experts` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--kt-max-deferred-`<br/>`experts-per-token` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Double Sparsity ## Double Sparsity
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
@@ -289,6 +320,12 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--offload-prefetch-step` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--offload-prefetch-step` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--offload-mode` | `cpu` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--offload-mode` | `cpu` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Args for multi-item scoring
| Argument | Defaults | Options | A2 | A3 |
|----------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
| `--multi-item-scoring-delimiter` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Optimization/debug options ## Optimization/debug options
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
@@ -304,6 +341,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--enable-symm-mem` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-symm-mem` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-flashinfer-`<br/>`cutlass-moe-fp4-allgather` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--disable-flashinfer-`<br/>`cutlass-moe-fp4-allgather` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--enable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disable-outlines-`<br/>`disk-cache` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--disable-outlines-`<br/>`disk-cache` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disable-custom-`<br/>`all-reduce` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--disable-custom-`<br/>`all-reduce` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-mscclpp` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-mscclpp` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
@@ -319,6 +357,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--enable-torch-`<br/>`compile-debug-mode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--enable-torch-`<br/>`compile-debug-mode` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-piecewise-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-piecewise-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--piecewise-cuda-`<br/>`graph-tokens` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--piecewise-cuda-`<br/>`graph-tokens` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--piecewise-cuda-`<br/>`graph-compiler` | `eager` | ["eager", "inductor"] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--torch-compile-max-bs` | `32` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--torch-compile-max-bs` | `32` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--piecewise-cuda-`<br/>`graph-max-tokens` | `4096` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | `--piecewise-cuda-`<br/>`graph-max-tokens` | `4096` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--torchao-config` | `` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--torchao-config` | `` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
@@ -331,6 +370,7 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-memory-saver` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-memory-saver` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-draft-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
@@ -339,27 +379,31 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-return-`<br/>`routed-experts` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--scheduler-recv-`<br/>`interval` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--scheduler-recv-`<br/>`interval` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--numa-node` | `None` | List[int] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--numa-node` | `None` | List[int] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--rl-on-policy-target` | `None` | `fsdp` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-layerwise-`<br/>`nvtx-marker` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-layerwise-`<br/>`nvtx-marker` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-attn-tp-`<br/>`input-scattered` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-attn-tp-`<br/>`input-scattered` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-nsa-prefill-`<br/>`context-parallel` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-nsa-prefill-`<br/>`context-parallel` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-fused-qk-`<br/>`norm-rope` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Forward hooks ## Dynamic batch tokenizer
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
|-------------------|----------|-----------------|:--------------------------------------:|:--------------------------------------:| |--------------------------------------------------|----------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--forward-hooks` | `None` | Type: JSON list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-dynamic-`<br/>`batch-tokenizer` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dynamic-batch-`<br/>`tokenizer-batch-size` | `32` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dynamic-batch-`<br/>`tokenizer-batch-timeout` | `0.002` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Debug tensor dumps ## Debug tensor dumps
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
|--------------------------------------------------|----------|--------------------------------|:----------------------------------------:|:----------------------------------------:| |--------------------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
| `--debug-tensor-dump-`<br/>`output-folder` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--debug-tensor-dump-`<br/>`layers` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--debug-tensor-dump-`<br/>`input-file` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--debug-tensor-dump-`<br/>`input-file` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--debug-tensor-dump-`<br/>`inject` | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--debug-tensor-dump-`<br/>`inject` | `False` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-dynamic-`<br/>`batch-tokenizer` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dynamic-batch-`<br/>`tokenizer-batch-size` | `32` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dynamic-batch-`<br/>`tokenizer-batch-timeout` | `0.002` | Type: float | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## PD disaggregation ## PD disaggregation
@@ -373,18 +417,30 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--disaggregation-prefill-pp` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--disaggregation-prefill-pp` | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disaggregation-ib-device` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--disaggregation-ib-device` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disaggregation-decode-`<br/>`enable-offload-kvcache` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--disaggregation-decode-`<br/>`enable-offload-kvcache` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--disaggregation-decode-`<br/>`enable-fake-auto` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--num-reserved-decode-tokens` | `512` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--num-reserved-decode-tokens` | `512` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--disaggregation-decode-`<br/>`polling-interval` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--disaggregation-decode-`<br/>`polling-interval` | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Encode prefill disaggregation
| Argument | Defaults | Options | A2 | A3 |
|------------------------------|----------|---------|:--------------------------------------:|:--------------------------------------:|
| `--encoder-only` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--language-only` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--encoder-transfer-backend` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--encoder-urls` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Custom weight loader ## Custom weight loader
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
|---------------------------------------------------------------------|----------|--------------------------------|:----------------------------------------:|------------------------------------------| |-------------------------------------------------------------------------|----------|--------------------------------|:----------------------------------------:|------------------------------------------|
| `--custom-weight-loader` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--custom-weight-loader` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-backend` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-start-seed-via-transfer-engine` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## For PD-Multiplexing ## For PD-Multiplexing
@@ -394,8 +450,39 @@ If you want to know the meaning and usage of each parameter, click [Service Argu
| `--pdmux-config-path` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--pdmux-config-path` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--sm-group-num` | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--sm-group-num` | `8` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## For Multi-Modal
| Argument | Defaults | Options | A2 | A3 |
|-----------------------------------------------|----------|-------------------|------------------------------------------|------------------------------------------|
| `--mm-max-concurrent-calls` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mm-per-request-timeout` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-broadcast-mm-`<br/>`inputs-process` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mm-process-config` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mm-enable-dp-encoder` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--limit-mm-data-per-request` | `None` | Type: JSON / Dict | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## For checkpoint decryption
| Argument | Defaults | Options | A2 | A3 |
|---------------------------------|----------|---------|:--------------------------------------:|:--------------------------------------:|
| `--decrypted-config-file` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--decrypted-draft-config-file` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-prefix-mm-cache` | | | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## For deterministic inference ## For deterministic inference
| Argument | Defaults | Options | A2 | A3 | | Argument | Defaults | Options | A2 | A3 |
|-------------------------------------------|----------|--------------------------------|----------------------------------------|----------------------------------------| |-------------------------------------------|----------|--------------------------------|----------------------------------------|----------------------------------------|
| `--enable-deterministic-`<br/>`inference` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | `--enable-deterministic-`<br/>`inference` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## For registering hooks
| Argument | Defaults | Options | A2 | A3 |
|-------------------|----------|-----------------|----------------------------------------|----------------------------------------|
| `--forward-hooks` | `None` | Type: JSON list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Configuration file support
| Argument | Defaults | Options | A2 | A3 |
|------------|----------|---------|----------------------------------------|----------------------------------------|
| `--config` | | yaml | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
docs/platforms/ascend_npu_support_models.md
+24 -24
View File
@@ -9,7 +9,7 @@ You are welcome to enable various models based on your business requirements.
| Models | Model Family | A2 Supported | A3 Supported | | Models | Model Family | A2 Supported | A3 Supported |
|-------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:| |-------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
| DeepSeek V3/V3.1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | DeepSeek V3/V3.1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| DeepSeek V3.2 | DeepSeek | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | DeepSeek V3.2 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| DeepSeek R1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | DeepSeek R1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| DeepSeek V2 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | DeepSeek V2 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | Qwen3 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -26,9 +26,9 @@ You are welcome to enable various models based on your business requirements.
| microsoft/Phi-4-multimodal-instruct | Phi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | microsoft/Phi-4-multimodal-instruct | Phi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| allenai/OLMoE-1B-7B-0924 | OLMoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | allenai/OLMoE-1B-7B-0924 | OLMoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| stabilityai/stablelm-2-1_6b | StableLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | stabilityai/stablelm-2-1_6b | StableLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| CohereForAI/c4ai-command-r-v01 | Command-R | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | CohereForAI/c4ai-command-r-v01 | Command-R | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| huihui-ai/grok-2 | Grok | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | huihui-ai/grok-2 | Grok | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| ZhipuAI/chatglm2-6b | ChatGLM | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | ZhipuAI/chatglm2-6b | ChatGLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| xverse/XVERSE-MoE-A36B | XVERSE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | xverse/XVERSE-MoE-A36B | XVERSE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -50,44 +50,44 @@ You are welcome to enable various models based on your business requirements.
| Models | Model Family (Variants) | A2 Supported | A3 Supported | | Models | Model Family (Variants) | A2 Supported | A3 Supported |
|-----------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:| |-----------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:|
| Qwen2.5-VL-72B-Instruct-w8a8 | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Qwen2.5-VL-72B-Instruct-w8a8 | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-VL-8B-Instruct | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Qwen3-VL-8B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| google/gemma-3-4b-it | Gemma 3 (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | google/gemma-3-4b-it | Gemma 3 (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.1-24B | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.1-24B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| microsoft/Phi-4-multimodal-instruct | Phi-4-multimodal-instruct | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | microsoft/Phi-4-multimodal-instruct | Phi-4-multimodal-instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| XiaomiMiMo/MiMo-VL-7B-RL | MiMo-VL (7B) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | XiaomiMiMo/MiMo-VL-7B-RL | MiMo-VL (7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| AI-ModelScope/llava-v1.6-34b | LLaVA (v1.5 & v1.6) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | AI-ModelScope/llava-v1.6-34b | LLaVA (v1.5 & v1.6) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| lmms-lab/llava-next-72b | LLaVA-NeXT (8B, 72B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | lmms-lab/llava-next-72b | LLaVA-NeXT (8B, 72B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| lmms-lab/llava-onevision-qwen2-7b-ov | LLaVA-OneVision | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | lmms-lab/llava-onevision-qwen2-7b-ov | LLaVA-OneVision | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Kimi/Kimi-VL-A3B-Instruct | Kimi-VL (A3B) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Kimi/Kimi-VL-A3B-Instruct | Kimi-VL (A3B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ZhipuAI/GLM-4.5V | GLM-4.5V (106B) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | ZhipuAI/GLM-4.5V | GLM-4.5V (106B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| meta-llama/Llama-3.2-11B-Vision-Instruct | Llama 3.2 Vision (11B) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | meta-llama/Llama-3.2-11B-Vision-Instruct | Llama 3.2 Vision (11B) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Embedding Models ## Embedding Models
| Models | Model Family | A2 Supported | A3 Supported | | Models | Model Family | A2 Supported | A3 Supported |
|-------------------------------------------|--------------------------|------------------------------------------|:----------------------------------------:| |-------------------------------------------|--------------------------|------------------------------------------|:----------------------------------------:|
| intfloat/e5-mistral-7b-instruct | E5 (Llama/Mistral based) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | intfloat/e5-mistral-7b-instruct | E5 (Llama/Mistral based) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| iic/gte_Qwen2-1.5B-instruct | GTE-Qwen2 | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | iic/gte_Qwen2-1.5B-instruct | GTE-Qwen2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-Embedding-8B | Qwen3-Embedding | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | Qwen/Qwen3-Embedding-8B | Qwen3-Embedding | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Alibaba-NLP/gme-Qwen2-VL-2B-Instruct | GME (Multimodal) | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Alibaba-NLP/gme-Qwen2-VL-2B-Instruct | GME (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| AI-ModelScope/clip-vit-large-patch14-336 | CLIP | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | | AI-ModelScope/clip-vit-large-patch14-336 | CLIP | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| BAAI/bge-large-en-v1.5 | BGE | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | BAAI/bge-large-en-v1.5 | BGE | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Reward Models ## Reward Models
| Models | Model Family | A2 Supported | A3 Supported | | Models | Model Family | A2 Supported | A3 Supported |
|---------------------------------------------|---------------------------|----------------------------------------|:----------------------------------------:| |---------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:|
| Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| jason9693/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** | | jason9693/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Skywork/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** | | Skywork/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Rerank Models ## Rerank Models