diff --git a/docs_new/docs/advanced_features/pd_disaggregation.mdx b/docs_new/docs/advanced_features/pd_disaggregation.mdx index dc0ef981b..492834007 100644 --- a/docs_new/docs/advanced_features/pd_disaggregation.mdx +++ b/docs_new/docs/advanced_features/pd_disaggregation.mdx @@ -173,7 +173,7 @@ The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom me **`SGLANG_DISAGGREGATION_THREAD_POOL_SIZE`** Controls the total number of worker threads for KVCache transfer operations per TP rank - A dynamic value calculated by int(0.75 * os.cpu_count()) // 8), which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions + A dynamic value calculated by int(0.75 * os.cpu_count()) // 8, which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions **`SGLANG_DISAGGREGATION_QUEUE_SIZE`** @@ -458,48 +458,253 @@ export ASCEND_NPU_PHY_ID=xxx ``` -### Llama Single Node +### MIMO Single Node + +#### prefill ```bash -python -m sglang.launch_server \ - --model-path meta-llama/Llama-3.1-8B-Instruct \ - --disaggregation-mode prefill \ - --port 30000 \ - --disaggregation-transfer-backend ascend -python -m sglang.launch_server \ - --model-path meta-llama/Llama-3.1-8B-Instruct \ - --disaggregation-mode decode \ - --port 30001 \ - --base-gpu-id 1 \ - --disaggregation-transfer-backend ascend -python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000 +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 +export HCCL_BUFFSIZE=1600 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_PROFILING=0 +export SGLANG_NPU_PROFILING_STAGE="prefill" +export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 +export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 +export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export SGLANG_DEEPEP_BF16_DISPATCH=0 +export ASCEND_USE_FIA=1 + +python3 -m sglang.launch_server \ + --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \ + --attention-backend ascend \ + --device npu \ + --tp-size 8 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 \ + --trust-remote-code --port 10000 \ + --host 127.0.0.1 --max-running-requests 16 \ + --mem-fraction-static 0.8 \ + --disaggregation-mode prefill --disaggregation-transfer-backend ascend \ + --disaggregation-bootstrap-port 8996 \ + --base-gpu-id 0 \ + --disable-radix-cache \ + --disable-cuda-graph \ + --moe-a2a-backend deepep --deepep-mode normal \ + # 2>&1 | tee $SGLANG_LOG_PATH +``` + +#### decode + +``` bash +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +# export ASCEND_LAUNCH_BLOCKING=1 + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256 +export HCCL_BUFFSIZE=1600 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_PROFILING=0 +export SGLANG_NPU_PROFILING_STAGE="prefill" +export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 +export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 +export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export SGLANG_DEEPEP_BF16_DISPATCH=0 +export ASCEND_USE_FIA=1 + +export SGLANG_NPU_FUSED_MOE_MODE=2 + +python3 -m sglang.launch_server \ + --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \ + --attention-backend ascend \ + --device npu \ + --tp-size 8 --nnodes 1 --node-rank 0 \ + --trust-remote-code --port 10001 \ + --host 127.0.0.1 --max-running-requests 16 \ + --mem-fraction-static 0.8 \ + --disaggregation-mode decode --disaggregation-transfer-backend ascend \ + --disaggregation-bootstrap-port 8996 \ + --base-gpu-id 8 \ + --disable-radix-cache \ + --cuda-graph-bs 1 2 4 8 10 12 14 16 \ + --quantization modelslim \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --enable-multi-layer-eagle \ + --moe-a2a-backend deepep --deepep-mode low_latency \ + # 2>&1 | tee $SGLANG_LOG_PATH +``` + + +#### router + +```bash +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --prefill http://127.0.0.1:10000 \ + --decode http://127.0.0.1:10001 \ + --host 127.0.0.1 \ + --port 9903 \ + --health-check-interval-secs 3600 \ + --mini-lb \ ``` ### DeepSeek Multi-Node +#### Environment ```bash -# prefill 0 -python -m sglang.launch_server \ - --model-path deepseek-ai/DeepSeek-V3-0324 \ - --disaggregation-transfer-backend ascend \ - --disaggregation-mode prefill \ - --host ${local_ip} \ - --port 30000 \ - --trust-remote-code \ - --dist-init-addr ${prefill_master_ip}:5000 \ - --nnodes 1 \ - --node-rank 0 \ - --tp-size 16 -# decode 0 -python -m sglang.launch_server \ - --model-path deepseek-ai/DeepSeek-V3-0324 \ - --disaggregation-transfer-backend ascend \ - --disaggregation-mode decode \ - --host ${local_ip} \ - --port 30001 \ - --trust-remote-code \ - --dist-init-addr ${decode_master_ip}:5000 \ - --nnodes 1 \ - --node-rank 0 \ - --tp-size 16 +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH} +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +# IP set to p first node ip +export ASCEND_MF_STORE_URL="tcp://XXXXXX:24670" + +# p node IP +P_IP=('XXXXX') +# D node IP +D_IP=('XXXXX') + + +# enable mlapo +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ENABLE_MOE_NZ=1 +#export SGLANG_NPU_USE_MULTI_STREAM=1 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" +``` + + +#### prefill + +```bash +MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_BUFFSIZE=2600 + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.7 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 32 --context-length 8192 --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-tokens 10240 --moe-a2a-backend deepep --deepep-mode normal \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --dp-size 8 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 + NODE_RANK=$i + break + fi +done +``` + +#### decode + +```bash +MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken + +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export SGLANG_ENABLE_SPEC_V2=1 + export HCCL_BUFFSIZE=900 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112 + export TASK_QUEUE_ENABLE=1 + export HCCL_SOCKET_IFNAME=data0.3001 + export GLOO_SOCKET_IFNAME=data0.3001 + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \ + --mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \ + --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \ + --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \ + --load-balance-method round_robin + NODE_RANK=$i + break + fi +done +``` + + +#### router + +```bash +python -m sglang_router.launch_router --prefill ${P_IP}:8000 \ +--decode ${D_IP}:8001 \ +--host ${D_IP} --port 6688 \ +--pd-disaggregation \ +--health-check-interval-secs 3600 \ ``` diff --git a/docs_new/docs/basic_usage/send_request.mdx b/docs_new/docs/basic_usage/send_request.mdx index 15ce9b121..81981fc48 100644 --- a/docs_new/docs/basic_usage/send_request.mdx +++ b/docs_new/docs/basic_usage/send_request.mdx @@ -16,7 +16,7 @@ This notebook provides a quick-start guide to use SGLang in chat completions aft ```python Example from sglang.test.doc_patch import launch_server_cmd -from sglang.utils import wait_for_server, print_highlight, terminate_process +from sglang.utils import wait_for_server, terminate_process # This is equivalent to running the following command in your terminal # python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 @@ -46,7 +46,7 @@ curl -s http://localhost:{port}/v1/chat/completions \ """ response = json.loads(subprocess.check_output(curl_command, shell=True)) -print_highlight(response) +print(response) ``` ## Using Python Requests @@ -64,7 +64,7 @@ data = { } response = requests.post(url, json=data) -print_highlight(response.json()) +print(response.json()) ``` ## Using OpenAI Python Client @@ -84,7 +84,7 @@ response = client.chat.completions.create( temperature=0, max_tokens=64, ) -print_highlight(response) +print(response) ``` ### Streaming @@ -133,7 +133,7 @@ response = requests.post( }, ) -print_highlight(response.json()) +print(response.json()) ``` ### Streaming diff --git a/docs_new/docs/developer_guide/msprobe_debugging_guide.mdx b/docs_new/docs/developer_guide/msprobe_debugging_guide.mdx index 97b6b0dd3..c40600cf2 100644 --- a/docs_new/docs/developer_guide/msprobe_debugging_guide.mdx +++ b/docs_new/docs/developer_guide/msprobe_debugging_guide.mdx @@ -205,6 +205,8 @@ msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./gra After the comparison or build task finishes, a `compare_{timestamp}.vis.db` file is created under `graph_output`. +The dumped data can be used to visualize and analyze differences using tables or charts generated with visualization tools such as Matplotlib and Excel. + ##### Launch TensorBoard Start TensorBoard: diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx index c751c6fca..eab466f2d 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx @@ -15,8 +15,9 @@ Before optimizing the performance, you need to obtain accurate performance data, The msProf tool is used to collect and analyze key performance indicators of operators running on the Ascend AI Processor. You can use the output performance data to quickly locate the software and hardware performance bottlenecks of operators and improve the efficiency of operator performance analysis. ```bash -msprof op --kernel-name=xxxxx python3 test_xxxxx.py +msprof op python3 test_xxxxx.py ``` +The following is a case of using msprof for data collection. | Attribute | Value | |------|------| @@ -63,18 +64,68 @@ msprof op --kernel-name=xxxxx python3 test_xxxxx.py | aiv_icache_miss_rate | 0.005 | | cube_utilization(%) | 0 | +Below is the field-by-field breakdown of the operator performance record, aligned with the official specification. +### 1. Basic Identification Fields +| Field | Value | Definition (per official docs) | +|-------|-------|--------------------------------| +| Name | DequantSwigluQuant_int32_high_performance_100000000 | **Op Name**: Name of the fused operator (dequantization + SwiGLU activation + quantization), with an int32 high-performance implementation suffix. | +| Type | DequantSwigluQuant | **OP Type**: Functional category of the operator. | +| OP State | static | **OP State**: Indicates a static operator whose shape and scheduling logic are determined at compile time. | +| Accelerator Core | AI_VECTOR_CORE | **Task Type**: The operator runs on the AI Vector Core; other common types include AI_CORE (matrix computation core) and AI_CPU. | -The Task Duration field indicates the time consumed by each operator. You can sort the operators by Task Duration to find the operators that consume the most time, or sort the operators by Task Type to view the operators that consume the most time on the AI Core or AI CPU. +### 2. Timing & Scheduling Fields +| Field | Value | Definition (per official docs) | +|-------|-------|--------------------------------| +| Start Time(us) | 1774489226717521.715 | **Task Start Time**: Absolute start timestamp of the operator task on the device side, in microseconds. | +| Duration(us) | 102.824 | **Task Duration**: End-to-end total latency of the operator, including dispatch time, accelerator execution time, and completion response time, in microseconds. | +| Wait Time(us) | 0 | **Task Wait Time**: Time interval between the end of the previous task and the start of the current task. A value of 0 means no idle wait between task dispatches. | -For some operators, the execution time is too long. As a result, the metric data is inaccurate and no longer has reference value. Such data is set to N/A and is not displayed. +### 3. Core Configuration & Precision Fields +| Field | Value | Definition (per official docs) | +|-------|-------|--------------------------------| +| Block Dim | 36 | **Block Num**: Number of parallel thread blocks for the operator task, corresponding to Block Dim in the SIMT programming model. One AI Vector Core executes only one thread block at a time, so this value reflects the scale of occupied parallel compute resources. | +| Mix Block Dim | 0 | **Mix Block Num**: Number of blocks on the secondary accelerator if the operator runs on both AI Core and Vector Core. A value of 0 means the operator runs exclusively on AI_VECTOR_CORE with no hybrid core scheduling. | +| HF32 Eligible | NO | **HF32 Eligible**: Indicates whether the HF32 high-precision floating-point format is enabled; `NO` means it is not used. This field is reported only at the `--task-time=l1` collection level. | -Input Shapes set to an empty value indicates that when the format is "; ; ; ;", the current input is a scalar. Here, ";" serves as the separator for each dimension. The output dimension of the operator follows the same principle. +### 4. Input & Output Information +| Field | Value | Definition (per official docs) | +|-------|-------|--------------------------------| +| Input Shapes | 163840,1024;128,1024;16384;;;;128 | **Input Shapes**: Dimensions of each input tensor, separated by semicolons; empty values represent scalar inputs. Breakdown: 7 inputs with shapes `[163840,1024]`, `[128,1024]`, `[163840]`, 3 scalars, and `[128]`. | +| Input Data Types | INT32;FLOAT;FLOAT;DT_UNDEFINED;DT_UNDEFINED;DT_UNDEFINED;INT64 | **Input Data Types**: Data types of inputs, in the same order as input shapes. | +| Input Formats | ND;ND;ND;NULL;NULL;NULL;ND | **Input Formats**: Memory layout of inputs; `ND` stands for N-dimensional tensor format, `NULL` corresponds to scalar/undefined inputs. | +| Output Shapes | 163840,512;163840 | **Output Shapes**: Dimensions of two output tensors, separated by semicolons: `[163840,512]` and `[163840]`. | +| Output Data Types | INT8;FLOAT | **Output Data Types**: Output 1 is INT8 (quantized result), output 2 is FLOAT. | +| Output Formats | ND;ND | **Output Formats**: Both outputs use standard ND layout. | +| Context ID | N/A | **Context ID**: Identifier for sub-tasks at Sub Task granularity; N/A means no sub-task splitting for this operator. | -- **Task Duration (us)**: Time required for running a task, including the time for scheduling a task to the accelerator, execution time on the accelerator, and response end time. The unit is μs. +### 5. AI Core Performance Metrics (aic_* series) +| Field | Value | Definition (per official docs) | +|-------|-------|--------------------------------| +| aicore_time(us) | 0 | Theoretical execution time on AI Core, in microseconds. | +| aic_total_cycles | 0 | Total execution cycles on AI Core. | +| aic_mac_time(us) / aic_mac_ratio | 0 / 0 | Latency and cycle ratio of cube (matrix multiplication) instructions. | +| aic_scalar_time(us) / aic_scalar_ratio | 0 / 0 | Latency and cycle ratio of scalar instructions. | +| aic_mte1_time(us) / aic_mte1_ratio | 0 / 0 | Latency and cycle ratio of L1→L0A/L0B data move instructions. | +| aic_mte2_time(us) / aic_mte2_ratio | 0 / 0 | Latency and cycle ratio of DDR→AICORE read move instructions. | +| aic_fixpipe_time(us) / aic_fixpipe_ratio | 0 / 0 | Latency and cycle ratio of L0C→OUT/L1 move instructions. | +| aic_icache_miss_rate | 0 | Instruction cache miss rate of AI Core. | -- **Task Wait Time (us)**: Interval between the end time of the previous task and the start time of the current task. The unit is μs. +### 6. AI Vector Core Performance Metrics (aiv_* series) +| Field | Value | Definition & Interpretation | +|-------|-------|------------------------------| +| aiv_time(us) | 59.128 | **aiv_time**: Theoretical execution time on Vector Core under ideal conditions (all blocks scheduled simultaneously with equal duration). In practice, this value is slightly smaller than real execution time due to staggered block startup. | +| aiv_total_cycles | 3512188 | **aiv_total_cycles**: Total cycles executed on Vector Core, summed across all blocks. | +| aiv_vec_time(us) / aiv_vec_ratio | 36.708 / 0.621 | Latency (us) and cycle ratio (62.1%) of vector computation instructions. Vector operations are the core compute workload of this operator. | +| aiv_scalar_time(us) / aiv_scalar_ratio | 41.403 / 0.7 | Latency (us) and cycle ratio (70%) of scalar instructions. The sum exceeds 100% because scalar and vector pipelines run in parallel with independent cycle counters. | +| aiv_mte2_time(us) / aiv_mte2_ratio | 11.975 / 0.203 | Latency and cycle ratio (20.3%) of read memory move instructions (DDR/on-chip memory → Vector Core). | +| aiv_mte3_time(us) / aiv_mte3_ratio | 9.738 / 0.165 | Latency and cycle ratio (16.5%) of write memory move instructions (Vector Core → DDR/on-chip memory). | +| aiv_icache_miss_rate | 0.005 | Vector Core instruction cache miss rate of 0.5%, extremely low and indicates efficient instruction fetch. | + +### 7. Utilization Metrics +| Field | Value | Definition (per official docs) | +|-------|-------|--------------------------------| +| cube_utilization(%) | 0 | **cube_utilization**: Utilization rate of the matrix multiplication unit. The value is 0 because the operator is purely vector-based. | -- **Block Dim**: Number of blocks into which a task is divided, which corresponds to the number of cores used for running the task. If task_time is L0, this field is not collected and is displayed as 0. ## Optimization @@ -111,3 +162,4 @@ Must be vectorized. 3. If the offset is a negative number, the current triton-ascend considers it as a discrete memory access scenario. As a result, the performance severely deteriorates, and the data is read from the entire DMA block instead of being read in scalar mode. 4. The UB of the Ascend hardware requires that the size of the tail axis of the tensor can be exactly divided by 32 bytes. If the length of the tail axis is insufficient, the length of the tail axis is automatically supplemented. For example, the performance deteriorates exponentially due to automatic supplementation for the Tensor whose shape is (2048, 3). In this situation, you can perform the transposition operation to change the alignment axis to a lower dimension. In addition, the transposition operation is affected by the automatic supplement rule. Therefore, special skills are also required to avoid supplementation. 5. Use Double Buffer, parallelizes computation and data transfer. While computing one block of data, another block of data is being transferred to L1. +6. If hostbound behavior is severe, core binding can be used to address it.