From 2e0d2d4c18f6987a4e64801105ba5765b52d4111 Mon Sep 17 00:00:00 2001 From: loading66 <102725781+loading66@users.noreply.github.com> Date: Thu, 21 May 2026 19:08:10 +0800 Subject: [PATCH] [NPU][DOCS]Add best practice and benchmark result parameter description (#25875) --- .../ascend-npus/ascend_npu_best_practice.mdx | 530 +++++++++++++++++- .../ascend_npu_performance_testing.mdx | 105 ++++ 2 files changed, 622 insertions(+), 13 deletions(-) diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx index 2617008cb..647871e03 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx @@ -264,7 +264,7 @@ you encounter issues or have any questions, please [open an issue](https://githu BF16 Optimal Configuration - + Qwen3-32B Atlas 800I A3 2 @@ -274,7 +274,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-32B Atlas 800I A3 2 @@ -284,7 +284,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-8B Atlas 800I A3 1 @@ -294,7 +294,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-8B Atlas 800I A3 1 @@ -304,7 +304,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-8B Atlas 800I A3 1 @@ -314,7 +314,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-30B-A3B Atlas 800I A3 1 @@ -324,7 +324,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-30B-A3B Atlas 800I A3 1 @@ -334,7 +334,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-Next-A3B-Instruct Atlas 800I A3 2 @@ -344,7 +344,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-Next-A3B-Instruct Atlas 800I A3 2 @@ -354,7 +354,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-Next-A3B-Instruct Atlas 800I A3 1 @@ -364,7 +364,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-14B Atlas 800I A3 1 @@ -374,6 +374,36 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration + + Qwen3-27B + Atlas 800I A3 + 2 + PD Mixed + 3.5K+1.5K + 20ms + W8A8 INT8 + Optimal Configuration + + + Qwen3-27B + Atlas 800I A3 + 1 + PD Mixed + 16K+1K + 20ms + W8A8 INT8 + Optimal Configuration + + + Qwen3-27B + Atlas 800I A3 + 1 + PD Mixed + 64K+1K + 20ms + W8A8 INT8 + Optimal Configuration + @@ -533,7 +563,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-14B Atlas 800I A3 1 @@ -543,7 +573,7 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration - + Qwen3-8B Atlas 800I A3 1 @@ -553,6 +583,36 @@ you encounter issues or have any questions, please [open an issue](https://githu W8A8 INT8 Optimal Configuration + + Qwen3-27B + Atlas 800I A3 + 1 + PD Mixed + 3.5K+1.5K + 50ms + W8A8 INT8 + Optimal Configuration + + + Qwen3-27B + Atlas 800I A3 + 2 + PD Mixed + 16K+1K + 50ms + W8A8 INT8 + Optimal Configuration + + + Qwen3-27B + Atlas 800I A3 + 2 + PD Mixed + 64K+1K + 50ms + W8A8 INT8 + Optimal Configuration + @@ -4154,3 +4214,447 @@ We tested it based on the `RANDOM` dataset. ```bash Command python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 3500 --num-prompts 1 ``` + +### Qwen3-27B 3_5K-1_5K 20ms on A3 2 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 + +export ASCEND_LAUNCH_BLOCKING=1 +export STREAMS_PER_DEVICE=32 +export HCCL_BUFFSIZE=3000 +export HCCL_OP_EXPANSION_MODE="AIV" +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_PROFILING=0 +export SGLANG_DISAGGEGATION_WAITING_TIMEOUT=3600 +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 + +MODEL_PATH=xxx + +python -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --host 127.0.0.1 --port 6699 \ + --device npu \ + --tp-size 4\ + --trust-remote-code \ + --watchdog-timeout 9000 \ + --chunked-prefill-size -1 \ + --max-prefill-tokens 186000 \ + --enable-prefill-delayer \ + --prefill-delayer-max-delay-passes 200 \ + --disable-radix-cache \ + --mem-fraction-static 0.94 \ + --max-total-tokens 700000 \ + --max-running-requests 38 \ + --max-mamba-cache-size 200 \ + --quantization modelslim \ + --dtype bfloat16 \ + --mamba-ssm-dtype bfloat16 \ + --enable-multimodal \ + --mm-attention-backend ascend_attn \ + --cuda-graph-bs 1 2 4 8 12 18 24 32 34 36 38 \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 6699 --dataset-name random --max-concurrency 38 --num-prompts 152 --random-range-ratio 1 --random-output-len 1500 --random-input-len 3500 +``` + +### Qwen3-27B 16K-1K 20ms on A3 1 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 16K+1K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 + +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=8,9 + +MODEL_PATH=xxx + +sglang serve --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 2 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 65000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 32 --max-mamba-cache-size 32 \ + --mem-fraction-static 0.85 \ + --port 8001 \ + --cuda-graph-bs 2 3 4 5 6 \ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 --max-total-tokens 310000 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8001 --dataset-name random --max-concurrency 32 --num-prompts 128 --random-range-ratio 1 --random-output-len 1000 --random-input-len 16000 +``` + +### Qwen3-27B 64K-1K 20ms on A3 1 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 64K+1K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_PROFILING=1 +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=4,5 + +MODEL_PATH=xxx + +python -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 2 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 130000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 32 --max-mamba-cache-size 18 \ + --mem-fraction-static 0.5 \ + --port 8004 \ + --cuda-graph-bs 2 3 4 \ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 --max-total-tokens 280000 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8004 --dataset-name random --max-concurrency 9 --num-prompts 36 --random-range-ratio 1 --random-output-len 1000 --random-input-len 64000 +``` + +### Qwen3-27B 3_5K-1_5K 50ms on A3 1 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 + +MODEL_PATH=xxx + +python -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 2 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 60000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 48 --max-mamba-cache-size 60 \ + --mem-fraction-static 0.7 \ + --port 8000 \ + --cuda-graph-bs 2 8 16 32 48 \ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8000 --dataset-name random --max-concurrency 48 --num-prompts 192 --random-range-ratio 1 --random-output-len 1500 --random-input-len 3500 +``` + +### Qwen3-27B 16K-1K 50ms on A3 2 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 16K+1K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30 +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 + +MODEL_PATH=xxx + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 4 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 50000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 28 --max-mamba-cache-size 50 \ + --mem-fraction-static 0.7 \ + --port 8001 \ + --cuda-graph-bs 2 8 12 16 20 24 28\ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8001 --dataset-name random --max-concurrency 28 --num-prompts 152 --random-range-ratio 1 --random-output-len 1000 --random-input-len 16000 +``` + +### Qwen3-27B 64K-1K 50ms on A3 2 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 64K+1K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 + +MODEL_PATH=xxx + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 4 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 200000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 32 --max-mamba-cache-size 22 \ + --mem-fraction-static 0.5 \ + --port 9000 \ + --cuda-graph-bs 2 4 8 11 12 13 \ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 --max-total-tokens 850000 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 9000 --dataset-name random --max-concurrency 9 --num-prompts 36 --random-range-ratio 1 --random-output-len 1000 --random-input-len 64000 +``` diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx index 56c79e678..9c1d4c54a 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx @@ -351,6 +351,111 @@ Max ITL (ms): 2229.30 ================================================== ``` +#### SGLang Serving Benchmark Result — Complete Reference + +The output format is **hardcoded in `bench_serving.py`. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments. + +##### Test Configuration + + + + + + + + + + + + + + + + + + + + + + +
ParameterDescription
BackendThe serving backend under test (e.g., sglang, vllm).
Traffic request rateRequest generation rate in req/s. inf means maximum rate (concurrency-bounded). trace indicates trace timestamp mode. A fixed value enforces constant inter-arrival time.
Max request concurrencyMaximum number of concurrent requests from the client side. Displays not set when unspecified.
+ +##### Core Statistics & Throughput Metrics + + + + + + + + + + + + + + + + + + + + + + + + +
ParameterDescriptionFormat Specification
Successful requestsTotal number of successfully completed requests (HTTP 200, no generation errors).Integer, no decimal places
Benchmark duration (s)Total elapsed time from first request sent to last response fully received (seconds).2 decimal places
Total input tokensTotal number of input (prompt) tokens across all requests, counted by server-side tokenizer.Integer, no decimal places
Total input text tokensSame as Total input tokens. For multimodal inputs, this may differ.Integer, no decimal places
Total generated tokensTotal number of output tokens actually generated by the server (server-side tokenizer count).Integer, no decimal places
Total generated tokens (retokenized)Output text re-tokenized by the client using its own tokenizer. A large discrepancy indicates tokenizer mismatch or special tokens in output.Integer, no decimal places
Request throughput (req/s)Number of successful requests processed per second. Formula: Successful requests / Benchmark duration (s).2 decimal places
Input token throughput (tok/s)Number of input tokens processed per second. Formula: Total input tokens / Benchmark duration (s).2 decimal places
Output token throughput (tok/s)Number of output tokens generated per second. Formula: Total generated tokens / Benchmark duration (s).2 decimal places
Peak output token throughput (tok/s)Observed instantaneous peak output token generation rate during the test (computed over a sliding window).2 decimal places
Peak concurrent requestsMaximum number of requests being processed simultaneously on the server side. May exceed client-side Max request concurrency due to queueing.Integer, no decimal places
Total token throughput (tok/s)Sum of input and output token throughputs. Formula: Input token throughput + Output token throughput.2 decimal places
ConcurrencyAverage number of concurrent requests during the test (Little's Law). Formula: Sum of all E2E latencies / Benchmark duration.2 decimal places
+ +##### End-to-End Latency (E2E Latency) + + + + + + + + + +
StatisticDescriptionFormat
Mean E2E Latency (ms)Arithmetic mean2 decimal places
Median E2E Latency (ms)50th percentile2 decimal places
P90 E2E Latency (ms)90th percentile (90% of requests have latency ≤ this value)2 decimal places
P99 E2E Latency (ms)99th percentile2 decimal places
+ +##### Time to First Token (TTFT) + + + + + + + + +
StatisticDescriptionFormat
Mean TTFT (ms)Arithmetic mean2 decimal places
Median TTFT (ms)50th percentile2 decimal places
P99 TTFT (ms)99th percentile2 decimal places
+ +##### Time per Output Token (TPOT) – Excluding First Token +Formula: (E2E Latency - TTFT) / (Number of output tokens - 1) + + + + + + + + +
StatisticDescriptionFormat
Mean TPOT (ms)Arithmetic mean2 decimal places
Median TPOT (ms)50th percentile2 decimal places
P99 TPOT (ms)99th percentile2 decimal places
+ +##### Inter-Token Latency (ITL) + + + + + + + + + + +
StatisticDescriptionFormat
Mean ITL (ms)Average inter-token interval2 decimal places
Median ITL (ms)50th percentile inter-token interval2 decimal places
P95 ITL (ms)95th percentile (used to detect stalls)2 decimal places
P99 ITL (ms)99th percentile2 decimal places
Max ITL (ms)Maximum observed inter-token interval; useful for identifying severe blocking events2 decimal places
+ + ## 3. Online Service: Multimodal Model Test `Qwen/Qwen2.5-VL-7B-Instruct` for vision-language tasks.