[Doc] [NPU] GLM: Add GLM-5.3 and GLM-5.3-Flash deployment tutorials (#43372)
Co-authored-by: pengxingchen0810 <pengxingchen0810@noreply.gitcode.com>
This commit is contained in:
co-authored by
pengxingchen0810
parent
844d2b5dec
commit
4449d7ecb0
@@ -1134,6 +1134,8 @@
|
||||
"docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash",
|
||||
"docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1",
|
||||
"docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2",
|
||||
"docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3",
|
||||
"docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3_flash",
|
||||
"docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3",
|
||||
"docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6",
|
||||
"docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5",
|
||||
|
||||
@@ -0,0 +1,566 @@
|
||||
---
|
||||
title: "GLM-5.3"
|
||||
metatags:
|
||||
description: "Deploy GLM-5.3 on Ascend NPUs with SGLang, covering single-node, multi-node, and prefill-decode disaggregation deployments."
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
GLM-5.3 is a large-scale Mixture-of-Experts (MoE) language model in the GLM (General Language Model) series, developed
|
||||
by Zhipu AI. It uses DeepSeek Sparse Attention (DSA), supports multi-token prediction (MTP), and is designed for
|
||||
long-context text generation. GLM-5.3 is distinct from the separately released GLM-5.3-Flash model, which uses a
|
||||
hybrid sparse/linear-attention architecture.
|
||||
|
||||
This document demonstrates the deployment of GLM-5.3 on Ascend NPUs using SGLang, including single-node deployment,
|
||||
multi-node co-located (mixed) deployment, prefill-decode disaggregation, feature configuration, and performance optimization.
|
||||
The multi-node and prefill-decode disaggregation examples use a locally prepared `GLM-5.3-w8a8` checkpoint and a CANN 9.1.0 runtime.
|
||||
|
||||
## Supported features
|
||||
|
||||
| Feature | Example usage |
|
||||
|-------------------------------|---------------------------------------------------------------------------------------------------------|
|
||||
| Tensor Parallelism | `--tp-size 16` or `--tp-size 32` |
|
||||
| Data Parallelism | `--dp-size 16` (mixed) or `--dp-size 32` (decode) with `--enable-dp-attention` |
|
||||
| Expert Parallelism | `--ep-size 32` with `--moe-a2a-backend deepep` and `--deepep-mode low_latency` |
|
||||
| Pipeline Parallelism | `--pp-size 4` with `SGLANG_PP_LAYER_PARTITION=18,20,24,16` |
|
||||
| Prefill Context Parallelism | `--attn-cp-size 8` with `--enable-prefill-cp` and `--cp-strategy interleave` |
|
||||
| PD Disaggregation | `--disaggregation-mode prefill` / `decode` with `--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | `--chunked-prefill-size 65536` / `--max-prefill-tokens 280000` (mixed deployment) |
|
||||
| NPU Graph | `--cuda-graph-bs-decode 2` (mixed); `--disable-cuda-graph` (prefill); `--cuda-graph-bs-decode 1 2 3` (decode) |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN` with `--speculative-num-steps 3` and `--speculative-num-draft-tokens 4` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
<Note>
|
||||
The values in the **Example usage** column reflect the deployment scenarios described below and are not universal
|
||||
defaults. Some flags require a specific SGLang version or model checkpoint. For parameter details, see
|
||||
[Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
|
||||
recommended configurations, see [Best practices](#best-practices).
|
||||
</Note>
|
||||
|
||||
For feature compatibility and conflict information between features,
|
||||
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Environment
|
||||
|
||||
Before following this tutorial, complete the environment setup in the documents below:
|
||||
|
||||
- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
|
||||
It walks you through launching a container, starting the SGLang server, and sending a test request.
|
||||
- [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
|
||||
It covers version compatibility (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile,
|
||||
and recommended system settings (CPU power scheme, NUMA, swap).
|
||||
|
||||
The mixed and PD disaggregation scripts below require a CANN **9.1.0** environment, the corresponding ATB and customized
|
||||
operator libraries, a compatible SGLang source checkout, and working inter-node HCCL/Gloo connectivity. Ensure that
|
||||
all participating nodes use compatible software versions and can access the model weights. The PD topology additionally requires the configured MemFabric components.
|
||||
|
||||
### Model weights
|
||||
|
||||
Before downloading model weights, check the model size to reserve enough disk space.
|
||||
For multi-node deployment, download the weights to a shared directory accessible to all nodes or distribute an
|
||||
identical checkpoint to the same local path on every node.
|
||||
|
||||
- [GLM-5.3-BF16](https://www.modelscope.cn/models/ZhipuAI/GLM-5.3-BF16) (BF16, approximately 1.51 TB).
|
||||
- `GLM-5.3-w8a8` (a locally prepared W8A8 ModelSlim-quantized checkpoint used in the scripts below; set `MODEL_PATH`
|
||||
to the actual location). The checkpoint size and MTP-weight availability depend on the conversion output.
|
||||
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize model weights for Ascend devices.
|
||||
|
||||
<Info>
|
||||
The deployment examples below use a W8A8 checkpoint with `--quantization modelslim`. The PD disaggregation topology
|
||||
uses **two Atlas 800I A3 nodes for prefill and two additional A3 nodes for decode**. Make sure that the checkpoint
|
||||
contains the required MTP weights before enabling NEXTN speculative decoding.
|
||||
</Info>
|
||||
|
||||
For hardware specifications (memory per device, devices per node, and differences between A2 and A3),
|
||||
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).
|
||||
|
||||
## Installation
|
||||
|
||||
<Warning>
|
||||
Ensure sufficient disk space before pulling images. A CANN Docker image requires at least **30GB** of free space.
|
||||
CANN 9.1.0 base images do **not** by themselves include a ready-to-run SGLang environment: install a compatible
|
||||
SGLang build, TorchNPU, ATB, custom operators, and other required dependencies before running the deployment scripts.
|
||||
</Warning>
|
||||
|
||||
The commands below start CANN 9.1.0 base images. Follow
|
||||
[SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation)
|
||||
to install SGLang and the required runtime components inside the container. Use the matching image for your hardware.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Atlas 800I A3">
|
||||
|
||||
```bash Command
|
||||
docker pull quay.io/ascend/sglang:cann9.1.0-a3-B080
|
||||
|
||||
docker run -itd --shm-size=16g --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
quay.io/ascend/sglang:cann9.1.0-a3-B080
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="Atlas 800I A2">
|
||||
|
||||
```bash Command
|
||||
docker pull quay.io/ascend/sglang:cann9.1.0-a3-B080
|
||||
|
||||
docker run -itd --shm-size=16g --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
quay.io/ascend/sglang:cann9.1.0-a3-B080
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
<Tip>
|
||||
- If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
|
||||
container, for example: `-v /path/to/models:/models`.
|
||||
- Replace `${NAME}` with your container name or remove `--name` to use the default name.
|
||||
- The listed `/dev/davinci*` mappings follow the 16-device A3 example in the original tutorial. Remove mappings for
|
||||
devices that do not exist on your host, particularly when using A2 nodes.
|
||||
- If you already have a validated CANN 9.1.0 SGLang development image, you can use it instead of building from the base image.
|
||||
</Tip>
|
||||
|
||||
## Online service deployment
|
||||
|
||||
### Single-node deployment
|
||||
|
||||
The following single-node example is adapted from the GLM-5.2 tutorial and is provided as a
|
||||
**reference configuration**.
|
||||
|
||||
```shell
|
||||
#!/usr/bin/env bash
|
||||
# Reference only: adapted from the GLM-5.2 tutorial; not yet verified for GLM-5.3.
|
||||
# Update MODEL_PATH, SGLANG_ROOT, and the installed custom-op paths as required.
|
||||
MODEL_PATH='/path/to/GLM-5.3-w8a8'
|
||||
SGLANG_ROOT='/path/to/sglang'
|
||||
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
|
||||
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path "${MODEL_PATH}" \
|
||||
--attention-backend ascend --device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 4096 --max-prefill-tokens 8192 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 --port 8000 \
|
||||
--mem-fraction-static 0.75 --max-running-requests 32 \
|
||||
--served-model-name glm-5 \
|
||||
--quantization modelslim --dtype bfloat16 \
|
||||
--moe-a2a-backend deepep --deepep-mode auto \
|
||||
--disable-shared-experts-fusion \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4
|
||||
```
|
||||
|
||||
### Multi-node deployment
|
||||
|
||||
For co-located (mixed) serving across **two Atlas 800I A3 nodes**, both nodes participate in a single GLM-5.3
|
||||
inference instance. The following script is adapted from the supplied GLM-5.3 W8A8 multi-node configuration, using
|
||||
**TP=32**, **DP=16**, DeepEP `auto`, and NEXTN speculative decoding.
|
||||
|
||||
Modify the node IP addresses, model/source paths, and network interface, then run the **same script on both nodes**.
|
||||
The two nodes must use the same compatible runtime and be able to communicate with rank 0 via the distributed
|
||||
initialization address. In this script, the mixed-serving HTTP port is **6688** (not 8000).
|
||||
|
||||
```shell
|
||||
#!/usr/bin/env bash
|
||||
# GLM-5.3 W8A8: two-node co-located (mixed) deployment (Atlas 800I A3).
|
||||
# Run the same script on both nodes after replacing the variables below.
|
||||
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# IPS: IP addresses of two nodes (rank 0 first)
|
||||
# NIC: network interface name for HCCL and Gloo
|
||||
# MODEL_PATH: GLM-5.3 W8A8 weight directory
|
||||
# SGLANG_ROOT: compatible SGLang source checkout
|
||||
# ============================================================
|
||||
IPS=('<node-0-ip>' '<node-1-ip>')
|
||||
NIC='<network-interface>'
|
||||
MODEL_PATH='/path/to/GLM-5.3-w8a8'
|
||||
SGLANG_ROOT='/path/to/sglang'
|
||||
|
||||
# High-performance CPU settings (requires sufficient permissions).
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
|
||||
export LD_LIBRARY_PATH="/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH:-}"
|
||||
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
|
||||
|
||||
# Keep the allocator configuration disabled as in the supplied script.
|
||||
# export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_SOCKET_IFNAME="${NIC}"
|
||||
export GLOO_SOCKET_IFNAME="${NIC}"
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
|
||||
# Optional tuning flags from the original script (disabled by default).
|
||||
# export SGLANG_NPU_USE_MLAPO=1
|
||||
# export HCCL_OP_EXPANSION_MODE=AIV
|
||||
# export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
# export SGLANG_USE_AG_AFTER_QLORA=1
|
||||
# export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
|
||||
|
||||
# Match the configured node IP against all local IPv4 addresses.
|
||||
LOCAL_IPS=( $(hostname -I) )
|
||||
has_local_ip() {
|
||||
local wanted="$1" local_ip
|
||||
for local_ip in "${LOCAL_IPS[@]}"; do
|
||||
[[ "${wanted}" == "${local_ip}" ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
for i in "${!IPS[@]}"; do
|
||||
if has_local_ip "${IPS[$i]}"; then
|
||||
echo "Starting GLM-5.3 mixed deployment on ${IPS[$i]} (rank ${i})"
|
||||
export HCCL_BUFFSIZE=2000
|
||||
exec python3 -m sglang.launch_server \
|
||||
--model-path "${MODEL_PATH}" \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--dist-init-addr "${IPS[0]}:5000" \
|
||||
--tp-size 32 --nnodes 2 --node-rank "$i" \
|
||||
--dp-size 16 --enable-dp-attention \
|
||||
--chunked-prefill-size 65536 --max-prefill-tokens 280000 \
|
||||
--trust-remote-code \
|
||||
--host "${IPS[$i]}" \
|
||||
--mem-fraction-static 0.68 \
|
||||
--port 6688 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-bs-decode 2 \
|
||||
--max-running-requests 32 \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep --deepep-mode auto \
|
||||
--load-balance-method round_robin \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4
|
||||
fi
|
||||
done
|
||||
|
||||
echo "ERROR: Neither IPS address matches this host: ${LOCAL_IPS[*]}" >&2
|
||||
exit 1
|
||||
```
|
||||
|
||||
### Prefill-decode disaggregation deployment
|
||||
|
||||
PD disaggregation splits the prefill and decode stages onto separate node groups, reducing interference between
|
||||
long-prompt processing and autoregressive generation. This configuration uses **four Atlas 800I A3 nodes**:
|
||||
|
||||
- **Prefill group:** 2 nodes, `TP=8`, `PP=4`, `DP=1`, and prefill context parallelism `CP=8`. It runs DeepEP in `normal`
|
||||
mode and disables NPU graph capture.
|
||||
- **Decode group:** 2 nodes, `TP=32`, `DP=32`, `EP=32`. It runs DeepEP in `low_latency` mode with decode graph capture
|
||||
and NEXTN speculative decoding.
|
||||
|
||||
Run the following **same script on all four nodes**. The script selects the role and node rank by matching a configured
|
||||
IP address against the local network addresses. Update the model/source paths, the two prefill and two decode IP
|
||||
addresses, and the HCCL/Gloo interface before running it. The nodes must reach the prefill master for MemFabric
|
||||
bootstrap and should be started before the router.
|
||||
|
||||
```shell
|
||||
#!/usr/bin/env bash
|
||||
# GLM-5.3 W8A8: 2 prefill nodes + 2 decode nodes (Atlas 800I A3).
|
||||
# Run this same script on all four nodes, after replacing the placeholders below.
|
||||
# The IP addresses must match addresses returned by `hostname -I` on each node.
|
||||
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# P_IP: two prefill node IP addresses (rank 0 first)
|
||||
# D_IP: two decode node IP addresses (rank 0 first)
|
||||
# NIC: HCCL/Gloo network interface name
|
||||
# MODEL_PATH: local/shared GLM-5.3 W8A8 model directory
|
||||
# SGLANG_ROOT: SGLang source directory
|
||||
# Ensure MemFabric, ATB, and CANN 9.1.0 custom ops are installed.
|
||||
# ============================================================
|
||||
P_IP=('<prefill-node-0-ip>' '<prefill-node-1-ip>')
|
||||
D_IP=('<decode-node-0-ip>' '<decode-node-1-ip>')
|
||||
NIC='<network-interface>'
|
||||
MODEL_PATH='/path/to/GLM-5.3-w8a8'
|
||||
SGLANG_ROOT='/path/to/sglang'
|
||||
|
||||
# High-performance CPU configuration (requires sufficient permissions).
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
|
||||
export LD_LIBRARY_PATH="/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib:${LD_LIBRARY_PATH:-}"
|
||||
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
|
||||
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export ASCEND_MF_LOG_LEVEL=2
|
||||
export ASCEND_MF_STORE_URL="tcp://${P_IP[0]}:24707"
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
|
||||
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
|
||||
export HCCL_SOCKET_IFNAME="${NIC}"
|
||||
export GLOO_SOCKET_IFNAME="${NIC}"
|
||||
|
||||
# Test every local interface rather than assuming the relevant IP is first.
|
||||
LOCAL_IPS=( $(hostname -I) )
|
||||
has_local_ip() {
|
||||
local wanted="$1" local_ip
|
||||
for local_ip in "${LOCAL_IPS[@]}"; do
|
||||
[[ "${wanted}" == "${local_ip}" ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Prefill: two A3 nodes (32 NPU devices in total); TP=8, PP=4, CP=8.
|
||||
for i in "${!P_IP[@]}"; do
|
||||
if has_local_ip "${P_IP[$i]}"; then
|
||||
export HCCL_BUFFSIZE=1200
|
||||
export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
|
||||
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
|
||||
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
export ENABLE_PROFILING=0
|
||||
export SGLANG_PP_LAYER_PARTITION=18,20,24,16
|
||||
|
||||
exec python3 -m sglang.launch_server \
|
||||
--model-path "${MODEL_PATH}" \
|
||||
--disaggregation-mode prefill \
|
||||
--host "${P_IP[$i]}" --port 8000 \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--dist-init-addr "${P_IP[0]}:5000" \
|
||||
--trust-remote-code --nnodes 2 --node-rank "$i" \
|
||||
--mem-fraction-static 0.72 \
|
||||
--attention-backend ascend --device npu \
|
||||
--quantization modelslim --dtype bfloat16 \
|
||||
--tp-size 8 --pp-size 4 --dp-size 1 \
|
||||
--attn-cp-size 8 --enable-prefill-cp --cp-strategy interleave \
|
||||
--enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-dense-tp 1 --load-balance-method round_robin \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--max-running-requests 64 \
|
||||
--served-model-name glm-5 \
|
||||
--chunked-prefill-size 4096 --max-prefill-tokens 180000 \
|
||||
--moe-a2a-backend deepep --deepep-mode normal \
|
||||
--disable-shared-experts-fusion --disable-cuda-graph \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 1 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 2
|
||||
fi
|
||||
done
|
||||
|
||||
# Decode: two A3 nodes (32 NPU devices in total); TP=32, DP=32, EP=32.
|
||||
for i in "${!D_IP[@]}"; do
|
||||
if has_local_ip "${D_IP[$i]}"; then
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export DEEPEP_HCCL_BUFFSIZE=200
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
|
||||
export TASK_QUEUE_ENABLE=0
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
|
||||
exec python3 -m sglang.launch_server \
|
||||
--model-path "${MODEL_PATH}" \
|
||||
--disaggregation-mode decode \
|
||||
--host "${D_IP[$i]}" --port 8003 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr "${D_IP[0]}:5000" \
|
||||
--nnodes 2 --node-rank "$i" \
|
||||
--tp-size 32 --dp-size 32 --enable-dp-attention --ep-size 32 \
|
||||
--mem-fraction-static 0.97 --max-running-requests 32 \
|
||||
--attention-backend ascend --device npu \
|
||||
--quantization modelslim --dtype bfloat16 \
|
||||
--served-model-name glm-5 \
|
||||
--moe-a2a-backend deepep --deepep-mode low_latency \
|
||||
--cuda-graph-bs-decode 1 2 3 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 --context-length 180000 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--disable-shared-experts-fusion \
|
||||
--load-balance-method round_robin \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4
|
||||
fi
|
||||
done
|
||||
|
||||
echo "ERROR: None of the P_IP/D_IP addresses matches this host: ${LOCAL_IPS[*]}" >&2
|
||||
exit 1
|
||||
```
|
||||
|
||||
Launch the router after the prefill and decode services are ready. The router example below preserves the
|
||||
GLM-5.2 document's PD routing syntax and uses the prefill/decode group master addresses.
|
||||
|
||||
```shell
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# P_MASTER_IP: first prefill node IP (prefill rank 0)
|
||||
# D_MASTER_IP: first decode node IP (decode rank 0)
|
||||
# ROUTER_HOST_IP: router listening IP
|
||||
# ============================================================
|
||||
P_MASTER_IP="<prefill-node-0-ip>"
|
||||
D_MASTER_IP="<decode-node-0-ip>"
|
||||
ROUTER_HOST_IP="<router-host-ip>"
|
||||
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy round_robin \
|
||||
--prefill http://${P_MASTER_IP}:8000 8998 \
|
||||
--decode http://${D_MASTER_IP}:8003 \
|
||||
--host "${ROUTER_HOST_IP}" \
|
||||
--port 6688
|
||||
```
|
||||
|
||||
<Note>
|
||||
The prefill and decode groups use separate `--dist-init-addr` endpoints. The `ASCEND_MF_STORE_URL` must point to the
|
||||
prefill rank-0 node. The `--max-prefill-tokens 180000` and `--context-length 180000` settings describe this example
|
||||
configuration, **not** the model's maximum supported context. For troubleshooting, turn off optional overlap or
|
||||
multi-stream settings one at a time to isolate compatibility issues.
|
||||
</Note>
|
||||
|
||||
## Functional verification
|
||||
|
||||
After the service is started, invoke the model by sending a prompt. For PD disaggregation, use the router address and
|
||||
port `6688`; for single-node co-located serving, use port `8000`, and for the two-node mixed deployment, use port `6688`.
|
||||
|
||||
```shell
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# HOST: router/server host IP address
|
||||
# PORT: 6688 for the PD router or mixed deployment; 8000 for single-node mode
|
||||
# ============================================================
|
||||
HOST="<router-or-server-ip>"
|
||||
PORT=6688
|
||||
|
||||
curl http://${HOST}:${PORT}/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"text": "What is the capital of France?",
|
||||
"sampling_params": {
|
||||
"max_new_tokens": 64,
|
||||
"temperature": 0
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Expected result: an HTTP 200 response containing the generated answer (for example, "Paris").
|
||||
|
||||
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For
|
||||
additional testing examples (Health Check, Generate, Chat Completions, and port usage),
|
||||
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).
|
||||
|
||||
## Accuracy evaluation
|
||||
|
||||
For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).
|
||||
|
||||
## Performance
|
||||
|
||||
For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).
|
||||
|
||||
## Best practices
|
||||
|
||||
### Best practice configuration reference
|
||||
|
||||
GLM-5.3 does not have a validated standalone best-practice page in this tutorial. The
|
||||
[Online service deployment](#online-service-deployment) section above includes a four-node PD disaggregation
|
||||
configuration and a two-node mixed deployment configuration based on supplied GLM-5.3 scripts. The single-node
|
||||
co-located example remains a GLM-5.2-derived reference. The configurations have not been independently verified on hardware.
|
||||
|
||||
The two-node mixed example uses `TP=32`, `DP=16`, `HCCL_BUFFSIZE=2000`, and `--cuda-graph-bs-decode 2`.
|
||||
Disable profiling when measuring uninstrumented service performance.
|
||||
|
||||
For this PD disaggregation configuration, prefill prioritizes long sequences using pipeline/context parallelism and
|
||||
DeepEP `normal`, while decode prioritizes low latency using DP/EP, DeepEP `low_latency`, NPU multi-stream,
|
||||
and NEXTN. Tune `--mem-fraction-static`, graph capture batch sizes, and concurrency based on observed HBM usage,
|
||||
TTFT, TPOT, and request throughput. Check that the weight conversion and MTP settings are compatible.
|
||||
|
||||
For the full catalog of optimization features and their parameter and compatibility details,
|
||||
see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).
|
||||
|
||||
## Performance tuning
|
||||
|
||||
For supported features, see [Supported features](#supported-features). For detailed optimization
|
||||
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).
|
||||
|
||||
## FAQ
|
||||
|
||||
For common environment, installation, and parameter issues, refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).
|
||||
+596
@@ -0,0 +1,596 @@
|
||||
---
|
||||
title: "GLM-5.3-Flash"
|
||||
metatags:
|
||||
description: "Deploy GLM-5.3-Flash with SGLang on Ascend A3 and A5 NPUs, including single-node and PD disaggregation modes."
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
GLM-5.3-Flash is a model in the GLM series. SGLang supports its inference on Ascend NPUs with DeepSeek Sparse
|
||||
Attention (DSA), multi-token prediction (MTP), and ModelSlim quantized weights.
|
||||
|
||||
This document demonstrates the deployment of GLM-5.3-Flash on Ascend NPUs using SGLang, including single-node
|
||||
prefill-decode mixed deployment, prefill-decode disaggregation, feature configuration, and performance tuning.
|
||||
The examples use W8A8 weights on Atlas 800I A3 and W8A8_MXFP8 weights on Ascend A5.
|
||||
|
||||
## Supported features
|
||||
|
||||
| Feature | Example usage |
|
||||
| --- | --- |
|
||||
| Tensor Parallelism | `--tp-size 16` |
|
||||
| Pipeline Parallelism | `--pp-size 4` |
|
||||
| Data Parallelism | `--dp-size 16` and `--enable-dp-attention` |
|
||||
| DeepEP | `--moe-a2a-backend deepep` and `--deepep-mode auto` |
|
||||
| PD Disaggregation | `--disaggregation-mode prefill` and `--disaggregation-transfer-backend ascend` |
|
||||
| Quantization | `--quantization modelslim` |
|
||||
| Chunked Prefill | `--chunked-prefill-size 8192` |
|
||||
| NPU Graph | `--cuda-graph-bs-decode 16` |
|
||||
| Speculative Decoding | `--speculative-algorithm NEXTN`<br/>`--speculative-num-steps 3`<br/>`--speculative-eagle-topk 1`<br/>`--speculative-num-draft-tokens 4` |
|
||||
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
|
||||
| DP LM Head | `--enable-dp-lm-head` |
|
||||
|
||||
<Note>
|
||||
The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
|
||||
mode, and workload. For parameter details, see
|
||||
[Feature descriptions](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-descriptions); for
|
||||
recommended configurations for each deployment scenario, see [Best practices](#best-practices).
|
||||
</Note>
|
||||
|
||||
For feature compatibility and conflicts, see
|
||||
[Feature Compatibility](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-compatibility).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Environment
|
||||
|
||||
Before following this tutorial, complete the environment setup in the documents below:
|
||||
|
||||
- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — the fastest way to get started.
|
||||
It walks you through launching the official container image, starting the SGLang server, and sending a test request.
|
||||
Recommended if you are new to SGLang on Ascend.
|
||||
- [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide.
|
||||
It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
|
||||
Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
|
||||
the environment instead of using the official image.
|
||||
|
||||
### Model weights
|
||||
|
||||
Before downloading model weights, check the model size to reserve enough disk space.
|
||||
For PD disaggregation, download the weights to a shared directory accessible to both nodes.
|
||||
|
||||
- [GLM-5.3-Flash-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8) (W8A8, 332.97GB, for Atlas 800I A3)
|
||||
- [GLM-5.3-Flash-w8a8-mxfp8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8-mxfp8) (W8A8_MXFP8, 338.12GB, for Ascend A5)
|
||||
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model. For quantization guidance, see
|
||||
[Quantization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_quantization).
|
||||
|
||||
The checkpoint directory must contain the model configuration, tokenizer files, quantized weights,
|
||||
`quant_model_description.json`, and the MTP weights required by NEXTN.
|
||||
Set `MODEL_PATH` to this directory inside the container.
|
||||
|
||||
<Info>
|
||||
A3 single-node deployment uses one full Atlas 800I A3 node: eight cards and 16 dies, with `--tp-size 16`.
|
||||
A5 single-node deployment uses eight NPU devices, with `--tp-size 8`.
|
||||
Each PD disaggregation configuration uses two nodes of the same platform: one for prefill and one for decode.
|
||||
</Info>
|
||||
|
||||
For hardware specifications and terminology, see
|
||||
[Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
|
||||
For the PD disaggregation configuration reference, see [Best practices](#best-practices).
|
||||
|
||||
## Installation
|
||||
|
||||
<Warning>
|
||||
Reserve enough disk space for both the container image and model weights before pulling the image.
|
||||
</Warning>
|
||||
|
||||
Use a SGLang image with the runtime components required by your target platform and deployment mode.
|
||||
Replace `IMAGE`, `NAME`, and `MODEL_DIR` below, then start the container.
|
||||
For A5 PD disaggregation, choose an image that supports DP attention.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Atlas 800I A3">
|
||||
|
||||
```bash Command
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# IMAGE: SGLang image for Ascend A3
|
||||
# NAME: container name
|
||||
# MODEL_DIR: model weights directory on the host
|
||||
# ============================================================
|
||||
|
||||
IMAGE="swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-glm5.3-flash"
|
||||
NAME=sglang-glm53-flash-a3
|
||||
MODEL_DIR=/path/to/model-weights
|
||||
|
||||
docker pull "$IMAGE"
|
||||
|
||||
docker run -itd --shm-size=16g --name "$NAME" \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
-v "${MODEL_DIR}:/models/GLM-5.3-Flash" \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
"$IMAGE"
|
||||
|
||||
docker exec -it "$NAME" bash
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="Ascend A5">
|
||||
|
||||
```bash Command
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# IMAGE: SGLang image for Ascend A5
|
||||
# NAME: container name
|
||||
# MODEL_DIR: model weights directory on the host
|
||||
# ============================================================
|
||||
|
||||
IMAGE="<your A5 image>"
|
||||
NAME=sglang-glm53-flash-a5
|
||||
MODEL_DIR=/path/to/model-weights
|
||||
|
||||
docker pull "$IMAGE"
|
||||
|
||||
docker run -itd --shm-size=16g --name "$NAME" \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
-v "${MODEL_DIR}:/models/GLM-5.3-Flash" \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
"$IMAGE"
|
||||
|
||||
docker exec -it "$NAME" bash
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
<Tip>
|
||||
- The examples mount the host checkpoint directory at `/models/GLM-5.3-Flash`. Set
|
||||
`MODEL_PATH=/models/GLM-5.3-Flash` in the launch scripts when using this mount.
|
||||
- Give each container its own `NAME`. For PD disaggregation, start a container on both nodes.
|
||||
- The examples use the SGLang package installed in the image and require no source-code path or `PYTHONPATH` override.
|
||||
</Tip>
|
||||
|
||||
## Online service deployment
|
||||
|
||||
### Single-node deployment
|
||||
|
||||
Prefill and decode run in the same instance. Select the script for your platform and run it inside the container.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Atlas 800I A3">
|
||||
|
||||
Use the ModelSlim W8A8 checkpoint on one Atlas 800I A3 node, with TP16.
|
||||
Replace `MODEL_PATH` with the checkpoint directory inside the container.
|
||||
|
||||
```shell
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# MODEL_PATH: path to the model weights directory
|
||||
# ============================================================
|
||||
|
||||
MODEL_PATH=/path/to/model-weights
|
||||
|
||||
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export HCCL_BUFFSIZE=1536
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path "$MODEL_PATH" \
|
||||
--quantization modelslim \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 \
|
||||
--chunked-prefill-size 4096 \
|
||||
--max-prefill-tokens 8192 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.78 \
|
||||
--page-size 128 \
|
||||
--served-model-name GLM-5.3-Flash \
|
||||
--cuda-graph-bs-decode 16 \
|
||||
--max-running-requests 16 \
|
||||
--load-balance-method round_robin \
|
||||
--enable-multimodal \
|
||||
--moe-a2a-backend deepep --deepep-mode auto \
|
||||
--speculative-draft-model-path "$MODEL_PATH" \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--watchdog-timeout 9000 \
|
||||
--host 127.0.0.1 --port 8810
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="Ascend A5">
|
||||
|
||||
Use the ModelSlim W8A8_MXFP8 checkpoint on one A5 node, with TP8 and no DP attention.
|
||||
Replace `MODEL_PATH` with the checkpoint directory inside the container.
|
||||
|
||||
```shell
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# MODEL_PATH: path to the model weights directory
|
||||
# ============================================================
|
||||
|
||||
MODEL_PATH=/path/to/model-weights
|
||||
|
||||
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
|
||||
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/memfabric_hybrid/set_env.sh
|
||||
|
||||
export SGLANG_NPU_PROFILING=0
|
||||
export ASCEND_LAUNCH_BLOCKING=1
|
||||
export HCCL_BUFFSIZE=256
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path "$MODEL_PATH" \
|
||||
--quantization modelslim \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 8 --nnodes 1 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--max-prefill-tokens 8192 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.83 \
|
||||
--page-size 64 \
|
||||
--served-model-name GLM-5.3-Flash \
|
||||
--load-format auto \
|
||||
--max-running-requests 16 \
|
||||
--pre-warm-nccl \
|
||||
--speculative-draft-model-path "$MODEL_PATH" \
|
||||
--speculative-draft-kv-cache-dtype bf16 \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 4 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 5 \
|
||||
--watchdog-timeout 1200 \
|
||||
--cuda-graph-bs-decode 1 8 64 \
|
||||
--host 127.0.0.1 --port 8810
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Prefill-decode disaggregation deployment
|
||||
|
||||
PD disaggregation places prefill and decode on separate nodes. Replace the model path, node IP addresses, and
|
||||
network interface names, then run the same script on both nodes. The script selects its role by matching the local
|
||||
IP address.
|
||||
|
||||
Ensure the worker HTTP ports, prefill bootstrap port, and MemFabric store port are reachable between the nodes.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Atlas 800I A3">
|
||||
|
||||
Use W8A8 weights on two Atlas 800I A3 nodes. The prefill instance uses TP4 / PP4 / DP4; the decode instance uses
|
||||
TP16 / PP1 / DP16 and NEXTN speculative decoding with three steps and four draft tokens.
|
||||
Run the CPU tuning commands with the required host privileges.
|
||||
|
||||
```shell
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# MODEL_PATH: path to the model weights directory
|
||||
# PREFILL_IP: prefill node IP address
|
||||
# DECODE_IP: decode node IP address
|
||||
# HCCL_SOCKET_IFNAME: network interface name for HCCL
|
||||
# GLOO_SOCKET_IFNAME: network interface name for Gloo
|
||||
# ============================================================
|
||||
|
||||
MODEL_PATH=/path/to/model-weights
|
||||
PREFILL_IP="<your prefill ip>"
|
||||
DECODE_IP="<your decode ip>"
|
||||
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export HCCL_SOCKET_IFNAME="<network-interface>"
|
||||
export GLOO_SOCKET_IFNAME="<network-interface>"
|
||||
export ASCEND_MF_STORE_URL="tcp://${PREFILL_IP}:24707"
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
|
||||
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
|
||||
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY
|
||||
export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${no_proxy:-}"
|
||||
export NO_PROXY="$no_proxy"
|
||||
LOCAL_IPS=" $(hostname -I) "
|
||||
|
||||
if [[ "$LOCAL_IPS" == *" $PREFILL_IP "* ]]; then
|
||||
export SGLANG_HOST_IP="$PREFILL_IP"
|
||||
export HCCL_BUFFSIZE=1200
|
||||
export DEEPEP_HCCL_BUFFSIZE=1200
|
||||
export DEEPEP_NORMAL_LONG_SEQ_ROUND=128
|
||||
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
export HCCL_IF_BASE_PORT=61000
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path "$MODEL_PATH" \
|
||||
--quantization modelslim \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--enable-multimodal \
|
||||
--served-model-name GLM-5.3-Flash \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--dist-init-addr "${PREFILL_IP}:5000" \
|
||||
--tp-size 4 --pp-size 4 --dp-size 4 --nnodes 1 \
|
||||
--enable-dp-attention \
|
||||
--enable-dp-lm-head \
|
||||
--moe-dense-tp 1 \
|
||||
--moe-a2a-backend deepep --deepep-mode normal \
|
||||
--load-balance-method round_robin \
|
||||
--mem-fraction-static 0.78 \
|
||||
--page-size 128 \
|
||||
--max-running-requests 16 \
|
||||
--chunked-prefill-size 4096 \
|
||||
--max-prefill-tokens 8192 \
|
||||
--watchdog-timeout 9000 \
|
||||
--host "$PREFILL_IP" --port 8810
|
||||
elif [[ "$LOCAL_IPS" == *" $DECODE_IP "* ]]; then
|
||||
export SGLANG_HOST_IP="$DECODE_IP"
|
||||
export HCCL_BUFFSIZE=512
|
||||
export DEEPEP_HCCL_BUFFSIZE=512
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export TASK_QUEUE_ENABLE=0
|
||||
export HCCL_IF_BASE_PORT=61010
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path "$MODEL_PATH" \
|
||||
--quantization modelslim \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--enable-multimodal \
|
||||
--served-model-name GLM-5.3-Flash \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--dist-init-addr "${DECODE_IP}:5000" \
|
||||
--tp-size 16 --pp-size 1 --dp-size 16 --nnodes 1 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep --deepep-mode low_latency \
|
||||
--load-balance-method round_robin \
|
||||
--mem-fraction-static 0.75 \
|
||||
--page-size 128 \
|
||||
--max-running-requests 32 \
|
||||
--cuda-graph-bs-decode 1 2 \
|
||||
--speculative-draft-model-path "$MODEL_PATH" \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--watchdog-timeout 9000 \
|
||||
--host "$DECODE_IP" --port 8811
|
||||
else
|
||||
echo "Local IP does not match PREFILL_IP or DECODE_IP: $LOCAL_IPS" >&2
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="Ascend A5">
|
||||
|
||||
Use W8A8_MXFP8 weights on two A5 nodes. Both instances use TP8 / PP1 / DP2 and NEXTN speculative decoding
|
||||
with four steps and five draft tokens. The nodes must be able to communicate over the `device_urma` transport.
|
||||
|
||||
```shell
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# MODEL_PATH: path to the model weights directory
|
||||
# PREFILL_IP: prefill node IP address
|
||||
# DECODE_IP: decode node IP address
|
||||
# ============================================================
|
||||
|
||||
MODEL_PATH=/path/to/model-weights
|
||||
PREFILL_IP="<your prefill ip>"
|
||||
DECODE_IP="<your decode ip>"
|
||||
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
|
||||
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash
|
||||
|
||||
export SGLANG_NPU_PROFILING=0
|
||||
export HCCL_BUFFSIZE=256
|
||||
export ASCEND_MF_STORE_URL="tcp://${PREFILL_IP}:24670"
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL=device_urma
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${no_proxy:-}"
|
||||
export NO_PROXY="$no_proxy"
|
||||
LOCAL_IPS=" $(hostname -I) "
|
||||
|
||||
if [[ "$LOCAL_IPS" == *" $PREFILL_IP "* ]]; then
|
||||
export SGLANG_HOST_IP="$PREFILL_IP"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path "$MODEL_PATH" \
|
||||
--quantization modelslim --load-format auto \
|
||||
--device npu --attention-backend ascend --trust-remote-code \
|
||||
--served-model-name GLM-5.3-Flash \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--tp-size 8 --pp-size 1 --dp-size 2 --enable-dp-attention --nnodes 1 \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 16384 \
|
||||
--mem-fraction-static 0.83 --page-size 64 --max-running-requests 16 \
|
||||
--speculative-draft-model-path "$MODEL_PATH" \
|
||||
--speculative-draft-kv-cache-dtype bf16 \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 4 \
|
||||
--speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
|
||||
--pre-warm-nccl --watchdog-timeout 1200 --cuda-graph-bs-decode 1 8 64 \
|
||||
--host "$PREFILL_IP" --port 8810
|
||||
elif [[ "$LOCAL_IPS" == *" $DECODE_IP "* ]]; then
|
||||
export SGLANG_HOST_IP="$DECODE_IP"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path "$MODEL_PATH" \
|
||||
--quantization modelslim --load-format auto \
|
||||
--device npu --attention-backend ascend --trust-remote-code \
|
||||
--served-model-name GLM-5.3-Flash \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--tp-size 8 --pp-size 1 --dp-size 2 --enable-dp-attention --nnodes 1 \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 16384 \
|
||||
--mem-fraction-static 0.83 --page-size 64 --max-running-requests 16 \
|
||||
--speculative-draft-model-path "$MODEL_PATH" \
|
||||
--speculative-draft-kv-cache-dtype bf16 \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 4 \
|
||||
--speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
|
||||
--pre-warm-nccl --watchdog-timeout 1200 --cuda-graph-bs-decode 1 8 64 \
|
||||
--host "$DECODE_IP" --port 8811
|
||||
else
|
||||
echo "Local IP does not match PREFILL_IP or DECODE_IP: $LOCAL_IPS" >&2
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
Launch the router after the prefill and decode services are ready. Use the same node IP addresses as in the worker
|
||||
scripts. Client requests go to the router on port 8000; the worker ports are 8810 and 8811, and the prefill bootstrap
|
||||
port is 8998.
|
||||
|
||||
```shell
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# PREFILL_IP: prefill node IP address
|
||||
# DECODE_IP: decode node IP address
|
||||
# ROUTER_HOST_IP: router node IP address
|
||||
# ============================================================
|
||||
|
||||
PREFILL_IP="<your prefill ip>"
|
||||
DECODE_IP="<your decode ip>"
|
||||
ROUTER_HOST_IP="<your router ip>"
|
||||
export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${ROUTER_HOST_IP},${no_proxy:-}"
|
||||
export NO_PROXY="$no_proxy"
|
||||
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill "http://${PREFILL_IP}:8810" 8998 \
|
||||
--decode "http://${DECODE_IP}:8811" \
|
||||
--host "$ROUTER_HOST_IP" --port 8000 \
|
||||
--policy round_robin \
|
||||
--worker-startup-timeout-secs 3600 \
|
||||
--request-timeout-secs 7200 \
|
||||
--disable-retries
|
||||
```
|
||||
|
||||
## Functional verification
|
||||
|
||||
After the service starts, send a prompt to verify inference. For mixed deployment, run the command inside the
|
||||
container with `HOST=127.0.0.1` and `PORT=8810`. For PD disaggregation, use the router IP address and `PORT=8000`.
|
||||
|
||||
```shell
|
||||
# ============================================================
|
||||
# Before running, update the following variables:
|
||||
# HOST: server or router host address
|
||||
# PORT: 8810 for mixed deployment; 8000 for the router
|
||||
# ============================================================
|
||||
|
||||
HOST=127.0.0.1
|
||||
PORT=8810
|
||||
|
||||
curl "http://${HOST}:${PORT}/generate" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"text": "What is the capital of France?",
|
||||
"sampling_params": {
|
||||
"max_new_tokens": 64,
|
||||
"temperature": 0
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Check for an HTTP 200 response with the generated answer. Once the server prints
|
||||
`The server is fired up and ready to roll!`, it is ready to accept requests.
|
||||
For Health Check, Generate, Chat Completions, and port usage examples, see
|
||||
[Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
|
||||
|
||||
## Accuracy evaluation
|
||||
|
||||
For evaluation methods and datasets, see
|
||||
[Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation).
|
||||
|
||||
## Performance
|
||||
|
||||
For benchmark methods and metric definitions, see
|
||||
[Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing).
|
||||
|
||||
For a generated shared-prefix benchmark workload, use approximately 128K input tokens and 1K output tokens.
|
||||
For mixed deployment, use the worker host and port 8810. For PD disaggregation, use the router host and port 8000.
|
||||
Set both `--model` and `--tokenizer` to the checkpoint path and use `--served-model-name GLM-5.3-Flash`.
|
||||
|
||||
## Best practices
|
||||
|
||||
### Best practice configuration reference
|
||||
|
||||
The [Online service deployment](#online-service-deployment) section above contains complete scripts for
|
||||
single-node mixed deployment and PD disaggregation.
|
||||
|
||||
For the optimization feature catalog and compatibility details, see
|
||||
[Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization).
|
||||
|
||||
## Performance tuning
|
||||
|
||||
For the feature overview, see [Supported features](#supported-features). For tuning parameters and feature
|
||||
compatibility, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization).
|
||||
|
||||
## FAQ
|
||||
|
||||
For environment, installation, and general parameter issues, see
|
||||
[Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/ascend_npu_faq).
|
||||
Reference in New Issue
Block a user