diff --git a/docs/platforms/ascend/ascend_npu_best_practice.md b/docs/platforms/ascend/ascend_npu_best_practice.md index d42e4ee25..7e395aa6c 100644 --- a/docs/platforms/ascend/ascend_npu_best_practice.md +++ b/docs/platforms/ascend/ascend_npu_best_practice.md @@ -41,6 +41,8 @@ you encounter issues or have any questions, please [open an issue](https://githu | Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 6K+1.5K | 17ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-6k-1_5k-17ms-on-a3-2-cards-mixed-mode) | | Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 1K+0.3K | 7ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-1k-0_3k-7ms-on-a3-1-cards-mixed-mode) | | Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 6K+1.5K | 9ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-6k-1_5k-9ms-on-a3-1-cards-mixed-mode) | +| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 6K+1.5K | 10ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-6k-1_5k-10ms-on-a3-1-cards-mixed-mode) | +| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1K+0.3K | 8ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-1k-0_3k-8ms-on-a3-1-cards-mixed-mode) | ### High Throughput @@ -2779,3 +2781,131 @@ We tested it based on the `RANDOM` dataset. ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 120 --random-range-ratio 1 ``` + +### Qwen3-30B-A3B 6K-1_5K 10ms on A3 1 Cards Mixed Mode + +Model: Qwen3-30B-A3B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 6K+1.5K + +TPOT: 10ms + +#### Model Deployment + +```shell +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_ENABLE_SPEC_V2=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 16 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx --speculative-draft-model-quantization unquant \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size -1 --max-prefill-tokens 35000 \ + --tp-size 2 --mem-fraction-static 0.6 --cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 16 --random-output-len 1500 --random-input-len 6144 --num-prompts 16 +``` + +### Qwen3-30B-A3B 1K-0_3K 8ms on A3 1 Cards Mixed Mode + +Model: Qwen3-30B-A3B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 1K+0.3K + +TPOT: 8ms + +#### Model Deployment + +```shell +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_ENABLE_SPEC_V2=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 8 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx --speculative-draft-model-quantization unquant \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size -1 --max-prefill-tokens 35000 \ + --tp-size 2 --mem-fraction-static 0.7 --cuda-graph-bs 1 2 3 4 5 6 7 8 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 8 --random-output-len 300 --random-input-len 1024 --num-prompts 8 +```