diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx index 38a4ac0aa..73e2377a1 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx @@ -423,16 +423,6 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - - Qwen3-235B-A22B - Atlas 800I A3 - 8 - PD Mixed - 2K+2K - 100ms - W8A8 INT8 - Optimal Configuration - Qwen3-235B-A22B Atlas 800I A3 @@ -1900,78 +1890,6 @@ We tested it based on the `RANDOM` dataset. python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 272 --random-input-len 3500 --random-output-len 1500 --num-prompts 1088 --random-range-ratio 1 ``` -### Qwen3-235B-A22B 2K-2K 100ms on A3 8 Cards Mixed Mode - -Model: Qwen3-235B-A22B-W8A8 - -Hardware: Atlas 800I A3 8Card - -DeployMode: PD Mixed - -Dataset: random - -Input Output Length: 2K+2K - -TPOT: 100ms - -#### Model Deployment - -```bash Command -echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor -sysctl -w vm.swappiness=0 -sysctl -w kernel.numa_balancing=0 -sysctl -w kernel.sched_migration_cost_ns=50000 - -unset https_proxy -unset http_proxy -unset HTTPS_PROXY -unset HTTP_PROXY -unset ASCEND_LAUNCH_BLOCKING -source /usr/local/Ascend/ascend-toolkit/set_env.sh -source /usr/local/Ascend/nnal/atb/set_env.sh -source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash -export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH - -export SGLANG_SET_CPU_AFFINITY=1 -export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True -export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 - -MODEL_PATH=xxx - -LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` -LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` - -echo "${LOCAL_HOST1}" -echo "${LOCAL_HOST2}" - -export HCCL_BUFFSIZE=1200 -export HCCL_SOCKET_IFNAME=lo -export GLOO_SOCKET_IFNAME=lo -export HCCL_OP_EXPANSION_MODE="AIV" -export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 -export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=144 - -python -m sglang.launch_server --model-path $MODEL_PATH \ - --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \ - --attention-backend ascend --device npu --quantization modelslim \ - --max-running-requests 576 --context-length 8192 --dtype bfloat16 \ - --chunked-prefill-size 32768 --max-prefill-tokens 458880 \ - --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ - --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ - --disable-radix-cache --moe-a2a-backend deepep --deepep-mode auto --speculative-draft-model-quantization unquant \ - --tp 16 --dp-size 16 --enable-dp-attention --enable-dp-lm-head --mem-fraction-static 0.84 --cuda-graph-bs 8 16 20 24 32 36 - -``` - -#### Benchmark - -We tested it based on the `RANDOM` dataset. - -```shell Command -python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 576 --random-input-len 2000 --random-output-len 2000 --num-prompts 576 --random-range-ratio 1 -``` - ### Qwen3-235B-A22B 2K-2K 50ms on A3 8 Cards Mixed Mode Model: Qwen3-235B-A22B-W8A8