diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx index 3fe88df70..c4bf6eb58 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx @@ -51,9 +51,14 @@ You can install SGLang using any of the methods below. Please go through `System - ### Obtain CANN Image + You can obtain the dependency of a specified version of CANN through an image. + + +CANN images and SGLang images are hosted at different registry addresses. Make sure to pull them from the correct location. + + ```bash Command # for Atlas 800I A3 and Ubuntu OS docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11 @@ -138,7 +143,7 @@ pip install -e python[all_npu] #### Obtain Image You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image. 1. Download SGLang image -```angular2html +```bash Command dockerhub: docker.io/lmsysorg/sglang:$tag # Main-based tag, change main to specific version like v0.5.6, # you can get image for specific version @@ -163,7 +168,6 @@ __Notice:__ `--privileged` and `--network=host` are required by RDMA, which is t __Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container. ```bash Command - alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx index 2ea91da4c..586421a1c 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx @@ -19,22 +19,22 @@ This document provides a list of commonly used environment variables and aims to SGLANG_NPU_USE_MLAPO - Adopts the MLAPO fusion operator in attention <br/> preprocessing stage of the MLA model. + Adopts the MLAPO fusion operator in attention
preprocessing stage of the MLA model. false SGLANG_USE_FIA_NZ - Reshapes KV Cache for FIA NZ format.<br/> SGLANG_USE_FIA_NZ must be enabled with SGLANG_NPU_USE_MLAPO + Reshapes KV Cache for FIA NZ format.
SGLANG_USE_FIA_NZ must be enabled with SGLANG_NPU_USE_MLAPO false SGLANG_NPU_USE_MULTI_STREAM - Enable dual-stream computation of shared experts <br/> and routing experts in DeepSeek models.<br/> Enable dual-stream computation in DeepSeek NSA Indexer. + Enable dual-stream computation of shared experts
and routing experts in DeepSeek models.
Enable dual-stream computation in DeepSeek NSA Indexer. false SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT - Disable cast model weight tensor to a specific NPU <br/> ACL format. + Disable cast model weight tensor to a specific NPU
ACL format. false @@ -58,22 +58,22 @@ This document provides a list of commonly used environment variables and aims to DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS - Enable ant-moving function in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank. + Enable ant-moving function in dispatch stage. Indicates
the number of tokens transmitted per round on each rank. 8192 DEEPEP_NORMAL_LONG_SEQ_ROUND - Enable ant-moving function in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank. + Enable ant-moving function in dispatch stage. Indicates
the number of rounds transmitted on each rank. 1 DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ - Enable ant-moving function in combine stage. <br/> The value 0 means disabled. + Enable ant-moving function in combine stage.
The value 0 means disabled. 0 MOE_ENABLE_TOPK_NEG_ONE - Needs to be enabled when the expert ID to be processed by <br/> DEEPEP contains -1. + Needs to be enabled when the expert ID to be processed by
DEEPEP contains -1. 0 @@ -97,7 +97,7 @@ This document provides a list of commonly used environment variables and aims to TASK_QUEUE_ENABLE - Used to control the optimization level of the dispatch queue<br/> about the task_queue operator. Detail + Used to control the optimization level of the dispatch queue
about the task_queue operator. Detail 1 @@ -112,12 +112,12 @@ This document provides a list of commonly used environment variables and aims to PYTORCH_NPU_ALLOC_CONF - Controls the behavior of the cache allocator. <br/>This variable changes memory usage and may cause performance fluctuations. Detail + Controls the behavior of the cache allocator.
This variable changes memory usage and may cause performance fluctuations. Detail ASCEND_MF_STORE_URL - The address of config store in MemFabric during PD separation, <br/>which is generally set to the IP address of the P primary node<br/> with an arbitrary port number. + The address of config store in MemFabric during PD separation,
which is generally set to the IP address of the P primary node
with an arbitrary port number. @@ -132,12 +132,12 @@ This document provides a list of commonly used environment variables and aims to HCCL_BUFFSIZE - Controls the size of the buffer area for shared data between two NPUs. <br/>The unit is MB, and the value must be greater than or equal to 1. Detail + Controls the size of the buffer area for shared data between two NPUs.
The unit is MB, and the value must be greater than or equal to 1. Detail 200 HCCL_SOCKET_IFNAME - Configures the name of the network card used by the Host <br/>during HCCL initialization. Detail + Configures the name of the network card used by the Host
during HCCL initialization. Detail diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx index 7a88a5e93..3b4545339 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx @@ -20,7 +20,7 @@ __Notice:__ The following commands are based on Atlas 800I A3 machines. If you a ```shell Command # For Atlas 800I A3 -export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-a3 +export IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx index f9fad5ad3..73a6b065a 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx @@ -7,13 +7,13 @@ metatags: ### Installation -The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it. +The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. ```bash Command #Atlas 800 A3 -docker pull quay.io/ascend/sglang:main-cann8.5.0-a3 +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3 #Atlas 800 A2 -docker pull quay.io/ascend/sglang:main-cann8.5.0-910b +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-910b #start container docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ @@ -42,7 +42,7 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -quay.io/ascend/sglang:${tag} +swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${tag} ``` ## Deployment @@ -53,6 +53,10 @@ Run the following script to execute online inference. #### Qwen3.5 397B + +Recommended model: `Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp` + + ```bash Command # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor @@ -97,6 +101,10 @@ python3 -m sglang.launch_server \ #### Qwen3.5 122B + +Recommended model: `Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp` + + ```bash Command # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor @@ -141,6 +149,10 @@ python3 -m sglang.launch_server \ #### Qwen3.5 35B + +Recommended model: `Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp` + + ```bash Command # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor @@ -185,6 +197,10 @@ python3 -m sglang.launch_server \ #### Qwen3.5 27B + +Recommended model: `Eco-Tech/Qwen3.5-27B-w8a8-mtp` + + ```bash Command # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx index e92622719..e4b3143d1 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx @@ -66,7 +66,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--model-loader-`
`extra-config` - {} + `{}` Type: str A2, A3 @@ -103,7 +103,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--model-impl` `auto` - auto, sglang,<br/> transformers + auto, sglang,
transformers A2, A3 @@ -248,7 +248,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--dtype` `auto` - auto,<br/> float16,<br/> bfloat16 + auto,
float16,
bfloat16 A2, A3 @@ -438,7 +438,7 @@ click [Server Arguments](../../advanced_features/server_arguments). --radix-eviction-policy lru - lru,<br/>lfu + lru,
lfu A2, A3 @@ -472,7 +472,7 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --abort-on-priority-<br/>when-disabled + --abort-on-priority-
when-disabled `False` bool flag
(set to enable) A2, A3 @@ -524,13 +524,13 @@ click [Server Arguments](../../advanced_features/server_arguments). Experimental - --attention-context-parallel-size<br/>--attn-cp-size + --attention-context-parallel-size
--attn-cp-size 1 Type: int; must be equal to --tp-size A2, A3 - --moe-data-parallel-size<br/>--moe-dp-size + --moe-data-parallel-size
--moe-dp-size 1 Type: int Planned @@ -578,13 +578,13 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --constrained-json-<br/>whitespace-pattern + --constrained-json-
whitespace-pattern None Type: str A2, A3 - --constrained-json-<br/>disable-any-whitespace + --constrained-json-
disable-any-whitespace False bool flag (set to enable) A2, A3 @@ -935,19 +935,19 @@ click [Server Arguments](../../advanced_features/server_arguments). --enable-cache-report False - bool flag<br/> (set to enable) + bool flag
(set to enable) A2, A3 --reasoning-parser `None` - deepseek-r1<br/>deepseek-v3<br/>glm45<br/>gpt-oss<br/>kimi<br/>qwen3<br/>qwen3-thinking<br/>step3 + deepseek-r1
deepseek-v3
glm45
gpt-oss
kimi
qwen3
qwen3-thinking
step3 A2, A3 --tool-call-parser None - llama3<br/> pythonic<br/> qwen<br/> qwen3_coder + llama3
pythonic
qwen
qwen3_coder A2, A3 @@ -987,7 +987,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--load-balance-method` auto - auto,<br/> round_robin,<br/> follow_bootstrap_room,<br/> total_requests,<br/> total_tokens + auto,
round_robin,
follow_bootstrap_room,
total_requests,
total_tokens A2, A3 @@ -1096,7 +1096,7 @@ click [Server Arguments](../../advanced_features/server_arguments). --enable-lora-overlap-loading False - Bool flag <br/>(set to enable) + Bool flag
(set to enable) A2, A3 @@ -1114,7 +1114,7 @@ click [Server Arguments](../../advanced_features/server_arguments). --lora-paths None - Type: List[str] /<br/> JSON objects + Type: List[str] /
JSON objects A2, A3 @@ -1132,13 +1132,13 @@ click [Server Arguments](../../advanced_features/server_arguments). --lora-eviction-policy lru - lru,<br/> fifo + lru,
fifo A2, A3 --lora-backend csgmv - triton,<br/>csgmv,<br/>ascend,<br/>torch_native + triton,
csgmv,
ascend,
torch_native A2, A3 @@ -1162,7 +1162,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--max-lora-chunk-size` `16` - 16, 32,<br/> 64, 128 + 16, 32,
64, 128 Special for GPU @@ -1209,7 +1209,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--sampling-backend` `None` - pytorch,<br/>ascend + pytorch,
ascend A2, A3 @@ -1227,19 +1227,19 @@ click [Server Arguments](../../advanced_features/server_arguments). `--nsa-prefill-backend` `flashmla_sparse` - flashmla_sparse,<br/> flashmla_decode,<br/>fa3,<br/> tilelang,<br/> aiter + flashmla_sparse,
flashmla_decode,
fa3,
tilelang,
aiter Special for GPU `--nsa-decode-backend` `fa3` - flashmla_prefill,<br/> flashmla_kv,<br/> fa3,<br/>tilelang,<br/> aiter + flashmla_prefill,
flashmla_kv,
fa3,
tilelang,
aiter Special for GPU `--fp8-gemm-backend` `auto` - auto,<br/> deep_gemm,<br/> flashinfer_trtllm,<br/>flashinfer_cutlass,<br/>flashinfer_deepgemm,<br/>cutlass,<br/> triton,<br/> aiter + auto,
deep_gemm,
flashinfer_trtllm,
flashinfer_cutlass,
flashinfer_deepgemm,
cutlass,
triton,
aiter Special for GPU @@ -1274,7 +1274,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--speculative-algorithm` `None` - EAGLE3,<br/> NEXTN + EAGLE3,
NEXTN A2, A3 @@ -1286,13 +1286,13 @@ click [Server Arguments](../../advanced_features/server_arguments). `--speculative-draft-model-`
`revision` `None` - Type: str,<br/> branch name,<br/> tag name,<br/> commit id + Type: str,
branch name,
tag name,
commit id A2, A3 `--speculative-draft-load-format` auto - auto,<br/> dummy + auto,
dummy A2, A3 @@ -1334,7 +1334,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--speculative-attention-`
`mode` `prefill` - prefill,<br/> decode + prefill,
decode A2, A3 @@ -1411,11 +1411,11 @@ click [Server Arguments](../../advanced_features/server_arguments). `--speculative-ngram-`
`match-type` `BFS` - BFS,<br/> PROB + BFS,
PROB Experimental. BFS uses recency-based expansion; PROB uses frequency-based expansion. - --speculative-ngram-<br/>max-trie-depth + --speculative-ngram-
max-trie-depth `18` Type: int Experimental @@ -1476,7 +1476,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--moe-a2a-backend` `none` - none,<br/> deepep,<br/> ascend_fuseep(It is incompatible with eplb) + none,
deepep,
ascend_fuseep(It is incompatible with eplb) A2, A3 @@ -1488,7 +1488,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--flashinfer-mxfp4-`
`moe-precision` `default` - default,<br/> bf16 + default,
bf16 Special for GPU @@ -1500,7 +1500,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--deepep-mode` `auto` - normal, <br/>low_latency,<br/> auto + normal,
low_latency,
auto A2, A3 @@ -1518,13 +1518,13 @@ click [Server Arguments](../../advanced_features/server_arguments). `--ep-dispatch-algorithm` `None` - static,<br/> dynamic,<br/> fake + static,
dynamic,
fake A2, A3 `--init-expert-location` `trivial` - trivial,<br/> <path.pt>,<br/> <path.json>,<br/> <json_string> + trivial,
<path.pt>,
<path.json>,
<json_string> A2, A3 @@ -1536,7 +1536,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--eplb-algorithm` deepseek - auto,<br/> deepseek + auto,
deepseek A2, A3 @@ -1546,31 +1546,31 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --eplb-rebalance-layers-<br/>per-chunk + --eplb-rebalance-layers-
per-chunk None Type: int A2, A3 - --eplb-min-rebalancing-<br/>utilization-threshold + --eplb-min-rebalancing-
utilization-threshold 1.0 Type: float A2, A3 - --expert-distribution-<br/>recorder-mode + --expert-distribution-
recorder-mode `None` - stat,<br/> stat_approx,<br/> per_pass,<br/> per_token + stat,
stat_approx,
per_pass,
per_token A2, A3 - --expert-distribution-<br/>recorder-buffer-size + --expert-distribution-
recorder-buffer-size None Type: int A2, A3 - --enable-expert-distribution-<br/>metrics + --enable-expert-distribution-
metrics False bool flag (set to enable) A2, A3 @@ -1624,7 +1624,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--mamba-ssm-dtype` `float32` - float32,<br/>bfloat16,<br/>float16 + float32,
bfloat16,
float16 A2, A3 @@ -1636,7 +1636,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--mamba-scheduler-strategy` `auto` - auto,<br/>no_buffer,<br/>extra_buffer + auto,
no_buffer,
extra_buffer A2, A3 @@ -1671,7 +1671,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--enable-hierarchical-`
`cache` `False` - bool flag<br/> (set to enable).<br/> Currently, mamba cache is not supported. + bool flag
(set to enable).
Currently, mamba cache is not supported. A2, A3 @@ -1695,29 +1695,29 @@ click [Server Arguments](../../advanced_features/server_arguments). --hicache-io-backend kernel - kernel_ascend,<br/> direct + kernel_ascend,
direct A2, A3 --hicache-mem-layout layer_first - page_first_direct,<br/> page_first_kv_split + page_first_direct,
page_first_kv_split A2, A3 - --hicache-storage-<br/>backend + --hicache-storage-
backend None file A2, A3 - --hicache-storage-<br/>prefetch-policy + --hicache-storage-
prefetch-policy timeout - best_effort,<br/> wait_complete,<br/> timeout + best_effort,
wait_complete,
timeout Special for GPU - --hicache-storage-<br/>backend-extra-config + --hicache-storage-
backend-extra-config None Type: str Special for GPU @@ -1834,7 +1834,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--offload-mode` `cpu` - cpu (DeepSeek only) <br/>meta (DeepSeek only) <br/>sharded_gpu (DeepSeek only) + cpu (DeepSeek only)
meta (DeepSeek only)
sharded_gpu (DeepSeek only) A2, A3 @@ -1956,13 +1956,13 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --disable-tokenizer-<br/>batch-decode + --disable-tokenizer-
batch-decode `False` bool flag
(set to enable) A2, A3 - --disable-custom-<br/>all-reduce + --disable-custom-
all-reduce `False` bool flag
(set to enable) Special for GPU @@ -1974,19 +1974,19 @@ click [Server Arguments](../../advanced_features/server_arguments). Special for GPU - --enable-torch-<br/>symm-mem + --enable-torch-
symm-mem `False` bool flag
(set to enable) Special for GPU - --disable-overlap<br/>-schedule + --disable-overlap
-schedule `False` bool flag
(set to enable) A2, A3 - --enable-mixed-<br/>chunk + --enable-mixed-
chunk `False` bool flag
(set to enable) A2, A3 @@ -2010,49 +2010,49 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --enable-two-<br/>batch-overlap + --enable-two-
batch-overlap `False` bool flag
(set to enable) Planned - --enable-single-<br/>batch-overlap + --enable-single-
batch-overlap `False` bool flag
(set to enable) A2, A3 - --tbo-token-<br/>distribution-threshold + --tbo-token-
distribution-threshold 0.48 Type: float Planned - --enable-torch-<br/>compile + --enable-torch-
compile False - bool flag<br/> (set to enable) + bool flag
(set to enable) A2, A3 - --enable-torch-<br/>compile-debug-mode + --enable-torch-
compile-debug-mode `False` bool flag
(set to enable) A2, A3 - --enforce-piecewise-<br/>cuda-graph + --enforce-piecewise-
cuda-graph `False` - bool flag<br/> (set to enable); <br/> Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported. + bool flag
(set to enable);
Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported. A2, A3 - --piecewise-cuda-<br/>graph-tokens + --piecewise-cuda-
graph-tokens None - Type: JSON<br/> list + Type: JSON
list A2, A3 - --piecewise-cuda-<br/>graph-compiler + --piecewise-cuda-
graph-compiler eager eager A2, A3 @@ -2064,7 +2064,7 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --piecewise-cuda-<br/>graph-max-tokens + --piecewise-cuda-
graph-max-tokens None Type: int A2, A3 @@ -2078,7 +2078,7 @@ click [Server Arguments](../../advanced_features/server_arguments). --enable-nan-detection False - bool flag<br/> (set to enable) + bool flag
(set to enable) A2, A3 @@ -2088,27 +2088,27 @@ click [Server Arguments](../../advanced_features/server_arguments). Special for GPU - --triton-attention-<br/>reduce-in-fp32 + --triton-attention-
reduce-in-fp32 `False` bool flag
(set to enable) Special for GPU - --triton-attention-<br/>num-kv-splits + --triton-attention-
num-kv-splits 8 Type: int Special for GPU - --triton-attention-<br/>split-tile-size + --triton-attention-
split-tile-size None Type: int Special for GPU - --delete-ckpt-<br/>after-loading + --delete-ckpt-
after-loading False - bool flag<br/> (set to enable) + bool flag
(set to enable) A2, A3 @@ -2118,13 +2118,13 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --enable-weights-<br/>cpu-backup + --enable-weights-
cpu-backup `False` bool flag
(set to enable) A2, A3 - --enable-draft-weights-<br/>cpu-backup + --enable-draft-weights-
cpu-backup `False` bool flag
(set to enable) A2, A3 @@ -2136,19 +2136,19 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --enable-custom-<br/>logit-processor + --enable-custom-
logit-processor `False` bool flag
(set to enable) A2, A3 - --flashinfer-mla-<br/>disable-ragged + --flashinfer-mla-
disable-ragged `False` bool flag
(set to enable) Special for GPU - --disable-shared-<br/>experts-fusion + --disable-shared-
experts-fusion True bool flag
(set to enable) A2, A3 @@ -2160,37 +2160,37 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --disable-chunked-<br/>prefix-cache + --disable-chunked-
prefix-cache True bool flag
(set to enable) A2, A3 - --disable-fast-<br/>image-processor + --disable-fast-
image-processor `False` bool flag
(set to enable) A2, A3 - --keep-mm-feature-<br/>on-device + --keep-mm-feature-
on-device `False` bool flag
(set to enable) A2, A3 - --enable-return-<br/>hidden-states + --enable-return-
hidden-states `False` bool flag
(set to enable) A2, A3 - --enable-return-<br/>routed-experts + --enable-return-
routed-experts `False` bool flag
(set to enable) A2, A3 - --scheduler-recv-<br/>interval + --scheduler-recv-
interval 1 Type: int A2, A3 @@ -2202,9 +2202,9 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --enable-deterministic-<br/>inference + --enable-deterministic-
inference False - bool flag<br/> (set to enable) + bool flag
(set to enable) Planned @@ -2367,7 +2367,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--disaggregation-mode` `null` - null,<br/> prefill,<br/> decode + null,
prefill,
decode A2, A3 @@ -2389,7 +2389,7 @@ click [Server Arguments](../../advanced_features/server_arguments). Special for GPU - --disaggregation-decode-<br/>enable-offload-kvcache + --disaggregation-decode-
enable-offload-kvcache False False A2, A3 @@ -2401,7 +2401,7 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - --disaggregation-decode-<br/>polling-interval + --disaggregation-decode-
polling-interval 1 Type: int A2, A3 @@ -2431,31 +2431,31 @@ click [Server Arguments](../../advanced_features/server_arguments). --enable-adaptive-dispatch-to-encoder `False` - bool flag<br/> (set to enable adaptively dispatch) + bool flag
(set to enable adaptively dispatch) A2, A3 --encoder-only `False` - bool flag<br/> (set to launch an encoder-only server) + bool flag
(set to launch an encoder-only server) A2, A3 --language-only False - bool flag<br/> (set to load weights for the language model only) + bool flag
(set to load weights for the language model only) A2, A3 --encoder-transfer-backend zmq_to_scheduler - zmq_to_scheduler, <br/> zmq_to_tokenizer,<br/> mooncake + zmq_to_scheduler,
zmq_to_tokenizer,
mooncake A2, A3 `--encoder-urls` `[]` - List[str]<br/> (List of encoder server urls) + List[str]
(List of encoder server urls) A2, A3 @@ -2526,7 +2526,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--remote-instance-weight-`
`loader-backend` `nccl` - transfer_engine, <br/> nccl + transfer_engine,
nccl A2, A3 @@ -2599,9 +2599,9 @@ click [Server Arguments](../../advanced_features/server_arguments). - --enable-broadcast-mm-<br/>inputs-process + --enable-broadcast-mm-
inputs-process False - bool flag<br/> (set to enable) + bool flag
(set to enable) A2, A3