diff --git a/docs/docs.json b/docs/docs.json index 6b7844b4b1..aa089919c5 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1134,6 +1134,8 @@ "docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash", "docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1", "docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2", + "docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3", + "docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3_flash", "docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3", "docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6", "docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5", diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3.mdx new file mode 100644 index 0000000000..35b604381a --- /dev/null +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3.mdx @@ -0,0 +1,566 @@ +--- +title: "GLM-5.3" +metatags: + description: "Deploy GLM-5.3 on Ascend NPUs with SGLang, covering single-node, multi-node, and prefill-decode disaggregation deployments." +--- + +## Introduction + +GLM-5.3 is a large-scale Mixture-of-Experts (MoE) language model in the GLM (General Language Model) series, developed +by Zhipu AI. It uses DeepSeek Sparse Attention (DSA), supports multi-token prediction (MTP), and is designed for +long-context text generation. GLM-5.3 is distinct from the separately released GLM-5.3-Flash model, which uses a +hybrid sparse/linear-attention architecture. + +This document demonstrates the deployment of GLM-5.3 on Ascend NPUs using SGLang, including single-node deployment, +multi-node co-located (mixed) deployment, prefill-decode disaggregation, feature configuration, and performance optimization. +The multi-node and prefill-decode disaggregation examples use a locally prepared `GLM-5.3-w8a8` checkpoint and a CANN 9.1.0 runtime. + +## Supported features + +| Feature | Example usage | +|-------------------------------|---------------------------------------------------------------------------------------------------------| +| Tensor Parallelism | `--tp-size 16` or `--tp-size 32` | +| Data Parallelism | `--dp-size 16` (mixed) or `--dp-size 32` (decode) with `--enable-dp-attention` | +| Expert Parallelism | `--ep-size 32` with `--moe-a2a-backend deepep` and `--deepep-mode low_latency` | +| Pipeline Parallelism | `--pp-size 4` with `SGLANG_PP_LAYER_PARTITION=18,20,24,16` | +| Prefill Context Parallelism | `--attn-cp-size 8` with `--enable-prefill-cp` and `--cp-strategy interleave` | +| PD Disaggregation | `--disaggregation-mode prefill` / `decode` with `--disaggregation-transfer-backend ascend` | +| Quantization | `--quantization modelslim` | +| Chunked Prefill | `--chunked-prefill-size 65536` / `--max-prefill-tokens 280000` (mixed deployment) | +| NPU Graph | `--cuda-graph-bs-decode 2` (mixed); `--disable-cuda-graph` (prefill); `--cuda-graph-bs-decode 1 2 3` (decode) | +| Speculative Decoding | `--speculative-algorithm NEXTN` with `--speculative-num-steps 3` and `--speculative-num-draft-tokens 4` | +| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | +| DP LM Head | `--enable-dp-lm-head` | + + +The values in the **Example usage** column reflect the deployment scenarios described below and are not universal +defaults. Some flags require a specific SGLang version or model checkpoint. For parameter details, see +[Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for +recommended configurations, see [Best practices](#best-practices). + + +For feature compatibility and conflict information between features, +see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). + +## Prerequisites + +### Environment + +Before following this tutorial, complete the environment setup in the documents below: + +- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. + It walks you through launching a container, starting the SGLang server, and sending a test request. +- [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. + It covers version compatibility (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, + and recommended system settings (CPU power scheme, NUMA, swap). + +The mixed and PD disaggregation scripts below require a CANN **9.1.0** environment, the corresponding ATB and customized +operator libraries, a compatible SGLang source checkout, and working inter-node HCCL/Gloo connectivity. Ensure that +all participating nodes use compatible software versions and can access the model weights. The PD topology additionally requires the configured MemFabric components. + +### Model weights + +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes or distribute an +identical checkpoint to the same local path on every node. + +- [GLM-5.3-BF16](https://www.modelscope.cn/models/ZhipuAI/GLM-5.3-BF16) (BF16, approximately 1.51 TB). +- `GLM-5.3-w8a8` (a locally prepared W8A8 ModelSlim-quantized checkpoint used in the scripts below; set `MODEL_PATH` + to the actual location). The checkpoint size and MTP-weight availability depend on the conversion output. +- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize model weights for Ascend devices. + + +The deployment examples below use a W8A8 checkpoint with `--quantization modelslim`. The PD disaggregation topology +uses **two Atlas 800I A3 nodes for prefill and two additional A3 nodes for decode**. Make sure that the checkpoint +contains the required MTP weights before enabling NEXTN speculative decoding. + + +For hardware specifications (memory per device, devices per node, and differences between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). + +## Installation + + +Ensure sufficient disk space before pulling images. A CANN Docker image requires at least **30GB** of free space. +CANN 9.1.0 base images do **not** by themselves include a ready-to-run SGLang environment: install a compatible +SGLang build, TorchNPU, ATB, custom operators, and other required dependencies before running the deployment scripts. + + +The commands below start CANN 9.1.0 base images. Follow +[SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) +to install SGLang and the required runtime components inside the container. Use the matching image for your hardware. + + + + +```bash Command +docker pull quay.io/ascend/sglang:cann9.1.0-a3-B080 + +docker run -itd --shm-size=16g --name ${NAME} \ +--privileged=true --net=host \ +-v /var/queue_schedule:/var/queue_schedule \ +-v /etc/ascend_install.info:/etc/ascend_install.info \ +-v /usr/local/sbin:/usr/local/sbin \ +-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ +-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ +--device=/dev/davinci0:/dev/davinci0 \ +--device=/dev/davinci1:/dev/davinci1 \ +--device=/dev/davinci2:/dev/davinci2 \ +--device=/dev/davinci3:/dev/davinci3 \ +--device=/dev/davinci4:/dev/davinci4 \ +--device=/dev/davinci5:/dev/davinci5 \ +--device=/dev/davinci6:/dev/davinci6 \ +--device=/dev/davinci7:/dev/davinci7 \ +--device=/dev/davinci8:/dev/davinci8 \ +--device=/dev/davinci9:/dev/davinci9 \ +--device=/dev/davinci10:/dev/davinci10 \ +--device=/dev/davinci11:/dev/davinci11 \ +--device=/dev/davinci12:/dev/davinci12 \ +--device=/dev/davinci13:/dev/davinci13 \ +--device=/dev/davinci14:/dev/davinci14 \ +--device=/dev/davinci15:/dev/davinci15 \ +--device=/dev/davinci_manager:/dev/davinci_manager \ +--device=/dev/hisi_hdc:/dev/hisi_hdc \ +--entrypoint=bash \ +quay.io/ascend/sglang:cann9.1.0-a3-B080 +``` + + + + +```bash Command +docker pull quay.io/ascend/sglang:cann9.1.0-a3-B080 + +docker run -itd --shm-size=16g --name ${NAME} \ +--privileged=true --net=host \ +-v /var/queue_schedule:/var/queue_schedule \ +-v /etc/ascend_install.info:/etc/ascend_install.info \ +-v /usr/local/sbin:/usr/local/sbin \ +-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ +-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ +--device=/dev/davinci0:/dev/davinci0 \ +--device=/dev/davinci1:/dev/davinci1 \ +--device=/dev/davinci2:/dev/davinci2 \ +--device=/dev/davinci3:/dev/davinci3 \ +--device=/dev/davinci4:/dev/davinci4 \ +--device=/dev/davinci5:/dev/davinci5 \ +--device=/dev/davinci6:/dev/davinci6 \ +--device=/dev/davinci7:/dev/davinci7 \ +--device=/dev/davinci8:/dev/davinci8 \ +--device=/dev/davinci9:/dev/davinci9 \ +--device=/dev/davinci10:/dev/davinci10 \ +--device=/dev/davinci11:/dev/davinci11 \ +--device=/dev/davinci12:/dev/davinci12 \ +--device=/dev/davinci13:/dev/davinci13 \ +--device=/dev/davinci14:/dev/davinci14 \ +--device=/dev/davinci15:/dev/davinci15 \ +--device=/dev/davinci_manager:/dev/davinci_manager \ +--device=/dev/hisi_hdc:/dev/hisi_hdc \ +--entrypoint=bash \ +quay.io/ascend/sglang:cann9.1.0-a3-B080 +``` + + + + + +- If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the + container, for example: `-v /path/to/models:/models`. +- Replace `${NAME}` with your container name or remove `--name` to use the default name. +- The listed `/dev/davinci*` mappings follow the 16-device A3 example in the original tutorial. Remove mappings for + devices that do not exist on your host, particularly when using A2 nodes. +- If you already have a validated CANN 9.1.0 SGLang development image, you can use it instead of building from the base image. + + +## Online service deployment + +### Single-node deployment + +The following single-node example is adapted from the GLM-5.2 tutorial and is provided as a +**reference configuration**. + +```shell +#!/usr/bin/env bash +# Reference only: adapted from the GLM-5.2 tutorial; not yet verified for GLM-5.3. +# Update MODEL_PATH, SGLANG_ROOT, and the installed custom-op paths as required. +MODEL_PATH='/path/to/GLM-5.3-w8a8' +SGLANG_ROOT='/path/to/sglang' + +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash +export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}" +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_BUFFSIZE=1000 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_USE_MULTI_STREAM=1 + +python3 -m sglang.launch_server \ + --model-path "${MODEL_PATH}" \ + --attention-backend ascend --device npu \ + --tp-size 16 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size 4096 --max-prefill-tokens 8192 \ + --trust-remote-code \ + --host 127.0.0.1 --port 8000 \ + --mem-fraction-static 0.75 --max-running-requests 32 \ + --served-model-name glm-5 \ + --quantization modelslim --dtype bfloat16 \ + --moe-a2a-backend deepep --deepep-mode auto \ + --disable-shared-experts-fusion \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 +``` + +### Multi-node deployment + +For co-located (mixed) serving across **two Atlas 800I A3 nodes**, both nodes participate in a single GLM-5.3 +inference instance. The following script is adapted from the supplied GLM-5.3 W8A8 multi-node configuration, using +**TP=32**, **DP=16**, DeepEP `auto`, and NEXTN speculative decoding. + +Modify the node IP addresses, model/source paths, and network interface, then run the **same script on both nodes**. +The two nodes must use the same compatible runtime and be able to communicate with rank 0 via the distributed +initialization address. In this script, the mixed-serving HTTP port is **6688** (not 8000). + +```shell +#!/usr/bin/env bash +# GLM-5.3 W8A8: two-node co-located (mixed) deployment (Atlas 800I A3). +# Run the same script on both nodes after replacing the variables below. + +# ============================================================ +# Before running, update the following variables: +# IPS: IP addresses of two nodes (rank 0 first) +# NIC: network interface name for HCCL and Gloo +# MODEL_PATH: GLM-5.3 W8A8 weight directory +# SGLANG_ROOT: compatible SGLang source checkout +# ============================================================ +IPS=('' '') +NIC='' +MODEL_PATH='/path/to/GLM-5.3-w8a8' +SGLANG_ROOT='/path/to/sglang' + +# High-performance CPU settings (requires sufficient permissions). +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash +export LD_LIBRARY_PATH="/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH:-}" +export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}" + +# Keep the allocator configuration disabled as in the supplied script. +# export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME="${NIC}" +export GLOO_SOCKET_IFNAME="${NIC}" +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40 +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + +# Optional tuning flags from the original script (disabled by default). +# export SGLANG_NPU_USE_MLAPO=1 +# export HCCL_OP_EXPANSION_MODE=AIV +# export SGLANG_NPU_USE_MULTI_STREAM=1 +# export SGLANG_USE_AG_AFTER_QLORA=1 +# export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 + +# Match the configured node IP against all local IPv4 addresses. +LOCAL_IPS=( $(hostname -I) ) +has_local_ip() { + local wanted="$1" local_ip + for local_ip in "${LOCAL_IPS[@]}"; do + [[ "${wanted}" == "${local_ip}" ]] && return 0 + done + return 1 +} + +for i in "${!IPS[@]}"; do + if has_local_ip "${IPS[$i]}"; then + echo "Starting GLM-5.3 mixed deployment on ${IPS[$i]} (rank ${i})" + export HCCL_BUFFSIZE=2000 + exec python3 -m sglang.launch_server \ + --model-path "${MODEL_PATH}" \ + --attention-backend ascend \ + --device npu \ + --dist-init-addr "${IPS[0]}:5000" \ + --tp-size 32 --nnodes 2 --node-rank "$i" \ + --dp-size 16 --enable-dp-attention \ + --chunked-prefill-size 65536 --max-prefill-tokens 280000 \ + --trust-remote-code \ + --host "${IPS[$i]}" \ + --mem-fraction-static 0.68 \ + --port 6688 \ + --served-model-name glm-5 \ + --cuda-graph-bs-decode 2 \ + --max-running-requests 32 \ + --quantization modelslim \ + --moe-a2a-backend deepep --deepep-mode auto \ + --load-balance-method round_robin \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 + fi +done + +echo "ERROR: Neither IPS address matches this host: ${LOCAL_IPS[*]}" >&2 +exit 1 +``` + +### Prefill-decode disaggregation deployment + +PD disaggregation splits the prefill and decode stages onto separate node groups, reducing interference between +long-prompt processing and autoregressive generation. This configuration uses **four Atlas 800I A3 nodes**: + +- **Prefill group:** 2 nodes, `TP=8`, `PP=4`, `DP=1`, and prefill context parallelism `CP=8`. It runs DeepEP in `normal` + mode and disables NPU graph capture. +- **Decode group:** 2 nodes, `TP=32`, `DP=32`, `EP=32`. It runs DeepEP in `low_latency` mode with decode graph capture + and NEXTN speculative decoding. + +Run the following **same script on all four nodes**. The script selects the role and node rank by matching a configured +IP address against the local network addresses. Update the model/source paths, the two prefill and two decode IP +addresses, and the HCCL/Gloo interface before running it. The nodes must reach the prefill master for MemFabric +bootstrap and should be started before the router. + +```shell +#!/usr/bin/env bash +# GLM-5.3 W8A8: 2 prefill nodes + 2 decode nodes (Atlas 800I A3). +# Run this same script on all four nodes, after replacing the placeholders below. +# The IP addresses must match addresses returned by `hostname -I` on each node. + +# ============================================================ +# Before running, update the following variables: +# P_IP: two prefill node IP addresses (rank 0 first) +# D_IP: two decode node IP addresses (rank 0 first) +# NIC: HCCL/Gloo network interface name +# MODEL_PATH: local/shared GLM-5.3 W8A8 model directory +# SGLANG_ROOT: SGLang source directory +# Ensure MemFabric, ATB, and CANN 9.1.0 custom ops are installed. +# ============================================================ +P_IP=('' '') +D_IP=('' '') +NIC='' +MODEL_PATH='/path/to/GLM-5.3-w8a8' +SGLANG_ROOT='/path/to/sglang' + +# High-performance CPU configuration (requires sufficient permissions). +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash +export LD_LIBRARY_PATH="/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib:${LD_LIBRARY_PATH:-}" +export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}" + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export ASCEND_MF_LOG_LEVEL=2 +export ASCEND_MF_STORE_URL="tcp://${P_IP[0]}:24707" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 +export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 +export HCCL_SOCKET_IFNAME="${NIC}" +export GLOO_SOCKET_IFNAME="${NIC}" + +# Test every local interface rather than assuming the relevant IP is first. +LOCAL_IPS=( $(hostname -I) ) +has_local_ip() { + local wanted="$1" local_ip + for local_ip in "${LOCAL_IPS[@]}"; do + [[ "${wanted}" == "${local_ip}" ]] && return 0 + done + return 1 +} + +# Prefill: two A3 nodes (32 NPU devices in total); TP=8, PP=4, CP=8. +for i in "${!P_IP[@]}"; do + if has_local_ip "${P_IP[$i]}"; then + export HCCL_BUFFSIZE=1200 + export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 + export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 + export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 + export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + export TASK_QUEUE_ENABLE=2 + export ENABLE_PROFILING=0 + export SGLANG_PP_LAYER_PARTITION=18,20,24,16 + + exec python3 -m sglang.launch_server \ + --model-path "${MODEL_PATH}" \ + --disaggregation-mode prefill \ + --host "${P_IP[$i]}" --port 8000 \ + --disaggregation-bootstrap-port 8998 \ + --dist-init-addr "${P_IP[0]}:5000" \ + --trust-remote-code --nnodes 2 --node-rank "$i" \ + --mem-fraction-static 0.72 \ + --attention-backend ascend --device npu \ + --quantization modelslim --dtype bfloat16 \ + --tp-size 8 --pp-size 4 --dp-size 1 \ + --attn-cp-size 8 --enable-prefill-cp --cp-strategy interleave \ + --enable-dp-attention --enable-dp-lm-head \ + --moe-dense-tp 1 --load-balance-method round_robin \ + --disaggregation-transfer-backend ascend \ + --max-running-requests 64 \ + --served-model-name glm-5 \ + --chunked-prefill-size 4096 --max-prefill-tokens 180000 \ + --moe-a2a-backend deepep --deepep-mode normal \ + --disable-shared-experts-fusion --disable-cuda-graph \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 1 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 2 + fi +done + +# Decode: two A3 nodes (32 NPU devices in total); TP=32, DP=32, EP=32. +for i in "${!D_IP[@]}"; do + if has_local_ip "${D_IP[$i]}"; then + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export DEEPEP_HCCL_BUFFSIZE=200 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 + export TASK_QUEUE_ENABLE=0 + export SGLANG_NPU_USE_MULTI_STREAM=1 + + exec python3 -m sglang.launch_server \ + --model-path "${MODEL_PATH}" \ + --disaggregation-mode decode \ + --host "${D_IP[$i]}" --port 8003 \ + --trust-remote-code \ + --dist-init-addr "${D_IP[0]}:5000" \ + --nnodes 2 --node-rank "$i" \ + --tp-size 32 --dp-size 32 --enable-dp-attention --ep-size 32 \ + --mem-fraction-static 0.97 --max-running-requests 32 \ + --attention-backend ascend --device npu \ + --quantization modelslim --dtype bfloat16 \ + --served-model-name glm-5 \ + --moe-a2a-backend deepep --deepep-mode low_latency \ + --cuda-graph-bs-decode 1 2 3 \ + --disaggregation-transfer-backend ascend \ + --watchdog-timeout 9000 --context-length 180000 \ + --tokenizer-worker-num 4 \ + --disable-shared-experts-fusion \ + --load-balance-method round_robin \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 + fi +done + +echo "ERROR: None of the P_IP/D_IP addresses matches this host: ${LOCAL_IPS[*]}" >&2 +exit 1 +``` + +Launch the router after the prefill and decode services are ready. The router example below preserves the +GLM-5.2 document's PD routing syntax and uses the prefill/decode group master addresses. + +```shell +# ============================================================ +# Before running, update the following variables: +# P_MASTER_IP: first prefill node IP (prefill rank 0) +# D_MASTER_IP: first decode node IP (decode rank 0) +# ROUTER_HOST_IP: router listening IP +# ============================================================ +P_MASTER_IP="" +D_MASTER_IP="" +ROUTER_HOST_IP="" + +python3 -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy round_robin \ + --prefill http://${P_MASTER_IP}:8000 8998 \ + --decode http://${D_MASTER_IP}:8003 \ + --host "${ROUTER_HOST_IP}" \ + --port 6688 +``` + + +The prefill and decode groups use separate `--dist-init-addr` endpoints. The `ASCEND_MF_STORE_URL` must point to the +prefill rank-0 node. The `--max-prefill-tokens 180000` and `--context-length 180000` settings describe this example +configuration, **not** the model's maximum supported context. For troubleshooting, turn off optional overlap or +multi-stream settings one at a time to isolate compatibility issues. + + +## Functional verification + +After the service is started, invoke the model by sending a prompt. For PD disaggregation, use the router address and +port `6688`; for single-node co-located serving, use port `8000`, and for the two-node mixed deployment, use port `6688`. + +```shell +# ============================================================ +# Before running, update the following variables: +# HOST: router/server host IP address +# PORT: 6688 for the PD router or mixed deployment; 8000 for single-node mode +# ============================================================ +HOST="" +PORT=6688 + +curl http://${HOST}:${PORT}/generate \ + -H "Content-Type: application/json" \ + -d '{ + "text": "What is the capital of France?", + "sampling_params": { + "max_new_tokens": 64, + "temperature": 0 + } + }' +``` + +Expected result: an HTTP 200 response containing the generated answer (for example, "Paris"). + +Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For +additional testing examples (Health Check, Generate, Chat Completions, and port usage), +see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). + +## Accuracy evaluation + +For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). + +## Performance + +For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). + +## Best practices + +### Best practice configuration reference + +GLM-5.3 does not have a validated standalone best-practice page in this tutorial. The +[Online service deployment](#online-service-deployment) section above includes a four-node PD disaggregation +configuration and a two-node mixed deployment configuration based on supplied GLM-5.3 scripts. The single-node +co-located example remains a GLM-5.2-derived reference. The configurations have not been independently verified on hardware. + +The two-node mixed example uses `TP=32`, `DP=16`, `HCCL_BUFFSIZE=2000`, and `--cuda-graph-bs-decode 2`. +Disable profiling when measuring uninstrumented service performance. + +For this PD disaggregation configuration, prefill prioritizes long sequences using pipeline/context parallelism and +DeepEP `normal`, while decode prioritizes low latency using DP/EP, DeepEP `low_latency`, NPU multi-stream, +and NEXTN. Tune `--mem-fraction-static`, graph capture batch sizes, and concurrency based on observed HBM usage, +TTFT, TPOT, and request throughput. Check that the weight conversion and MTP settings are compatible. + +For the full catalog of optimization features and their parameter and compatibility details, +see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). + +## Performance tuning + +For supported features, see [Supported features](#supported-features). For detailed optimization +guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). + +## FAQ + +For common environment, installation, and parameter issues, refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3_flash.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3_flash.mdx new file mode 100644 index 0000000000..30239d0dd5 --- /dev/null +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3_flash.mdx @@ -0,0 +1,596 @@ +--- +title: "GLM-5.3-Flash" +metatags: + description: "Deploy GLM-5.3-Flash with SGLang on Ascend A3 and A5 NPUs, including single-node and PD disaggregation modes." +--- + +## Introduction + +GLM-5.3-Flash is a model in the GLM series. SGLang supports its inference on Ascend NPUs with DeepSeek Sparse +Attention (DSA), multi-token prediction (MTP), and ModelSlim quantized weights. + +This document demonstrates the deployment of GLM-5.3-Flash on Ascend NPUs using SGLang, including single-node +prefill-decode mixed deployment, prefill-decode disaggregation, feature configuration, and performance tuning. +The examples use W8A8 weights on Atlas 800I A3 and W8A8_MXFP8 weights on Ascend A5. + +## Supported features + +| Feature | Example usage | +| --- | --- | +| Tensor Parallelism | `--tp-size 16` | +| Pipeline Parallelism | `--pp-size 4` | +| Data Parallelism | `--dp-size 16` and `--enable-dp-attention` | +| DeepEP | `--moe-a2a-backend deepep` and `--deepep-mode auto` | +| PD Disaggregation | `--disaggregation-mode prefill` and `--disaggregation-transfer-backend ascend` | +| Quantization | `--quantization modelslim` | +| Chunked Prefill | `--chunked-prefill-size 8192` | +| NPU Graph | `--cuda-graph-bs-decode 16` | +| Speculative Decoding | `--speculative-algorithm NEXTN`
`--speculative-num-steps 3`
`--speculative-eagle-topk 1`
`--speculative-num-draft-tokens 4` | +| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | +| DP LM Head | `--enable-dp-lm-head` | + + +The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment +mode, and workload. For parameter details, see +[Feature descriptions](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-descriptions); for +recommended configurations for each deployment scenario, see [Best practices](#best-practices). + + +For feature compatibility and conflicts, see +[Feature Compatibility](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-compatibility). + +## Prerequisites + +### Environment + +Before following this tutorial, complete the environment setup in the documents below: + +- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — the fastest way to get started. + It walks you through launching the official container image, starting the SGLang server, and sending a test request. + Recommended if you are new to SGLang on Ascend. +- [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide. + It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a + Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize + the environment instead of using the official image. + +### Model weights + +Before downloading model weights, check the model size to reserve enough disk space. +For PD disaggregation, download the weights to a shared directory accessible to both nodes. + +- [GLM-5.3-Flash-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8) (W8A8, 332.97GB, for Atlas 800I A3) +- [GLM-5.3-Flash-w8a8-mxfp8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8-mxfp8) (W8A8_MXFP8, 338.12GB, for Ascend A5) +- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model. For quantization guidance, see + [Quantization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_quantization). + +The checkpoint directory must contain the model configuration, tokenizer files, quantized weights, +`quant_model_description.json`, and the MTP weights required by NEXTN. +Set `MODEL_PATH` to this directory inside the container. + + +A3 single-node deployment uses one full Atlas 800I A3 node: eight cards and 16 dies, with `--tp-size 16`. +A5 single-node deployment uses eight NPU devices, with `--tp-size 8`. +Each PD disaggregation configuration uses two nodes of the same platform: one for prefill and one for decode. + + +For hardware specifications and terminology, see +[Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). +For the PD disaggregation configuration reference, see [Best practices](#best-practices). + +## Installation + + +Reserve enough disk space for both the container image and model weights before pulling the image. + + +Use a SGLang image with the runtime components required by your target platform and deployment mode. +Replace `IMAGE`, `NAME`, and `MODEL_DIR` below, then start the container. +For A5 PD disaggregation, choose an image that supports DP attention. + + + + +```bash Command +# ============================================================ +# Before running, update the following variables: +# IMAGE: SGLang image for Ascend A3 +# NAME: container name +# MODEL_DIR: model weights directory on the host +# ============================================================ + +IMAGE="swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-glm5.3-flash" +NAME=sglang-glm53-flash-a3 +MODEL_DIR=/path/to/model-weights + +docker pull "$IMAGE" + +docker run -itd --shm-size=16g --name "$NAME" \ + --privileged=true --net=host \ + -v /var/queue_schedule:/var/queue_schedule \ + -v /etc/ascend_install.info:/etc/ascend_install.info \ + -v /usr/local/sbin:/usr/local/sbin \ + -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ + -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ + -v "${MODEL_DIR}:/models/GLM-5.3-Flash" \ + --device=/dev/davinci0:/dev/davinci0 \ + --device=/dev/davinci1:/dev/davinci1 \ + --device=/dev/davinci2:/dev/davinci2 \ + --device=/dev/davinci3:/dev/davinci3 \ + --device=/dev/davinci4:/dev/davinci4 \ + --device=/dev/davinci5:/dev/davinci5 \ + --device=/dev/davinci6:/dev/davinci6 \ + --device=/dev/davinci7:/dev/davinci7 \ + --device=/dev/davinci8:/dev/davinci8 \ + --device=/dev/davinci9:/dev/davinci9 \ + --device=/dev/davinci10:/dev/davinci10 \ + --device=/dev/davinci11:/dev/davinci11 \ + --device=/dev/davinci12:/dev/davinci12 \ + --device=/dev/davinci13:/dev/davinci13 \ + --device=/dev/davinci14:/dev/davinci14 \ + --device=/dev/davinci15:/dev/davinci15 \ + --device=/dev/davinci_manager:/dev/davinci_manager \ + --device=/dev/hisi_hdc:/dev/hisi_hdc \ + --entrypoint=bash \ + "$IMAGE" + +docker exec -it "$NAME" bash +``` + + + + +```bash Command +# ============================================================ +# Before running, update the following variables: +# IMAGE: SGLang image for Ascend A5 +# NAME: container name +# MODEL_DIR: model weights directory on the host +# ============================================================ + +IMAGE="" +NAME=sglang-glm53-flash-a5 +MODEL_DIR=/path/to/model-weights + +docker pull "$IMAGE" + +docker run -itd --shm-size=16g --name "$NAME" \ + --privileged=true --net=host \ + -v /var/queue_schedule:/var/queue_schedule \ + -v /etc/ascend_install.info:/etc/ascend_install.info \ + -v /usr/local/sbin:/usr/local/sbin \ + -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ + -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ + -v "${MODEL_DIR}:/models/GLM-5.3-Flash" \ + --device=/dev/davinci0:/dev/davinci0 \ + --device=/dev/davinci1:/dev/davinci1 \ + --device=/dev/davinci2:/dev/davinci2 \ + --device=/dev/davinci3:/dev/davinci3 \ + --device=/dev/davinci4:/dev/davinci4 \ + --device=/dev/davinci5:/dev/davinci5 \ + --device=/dev/davinci6:/dev/davinci6 \ + --device=/dev/davinci7:/dev/davinci7 \ + --device=/dev/davinci_manager:/dev/davinci_manager \ + --device=/dev/hisi_hdc:/dev/hisi_hdc \ + --entrypoint=bash \ + "$IMAGE" + +docker exec -it "$NAME" bash +``` + + + + + +- The examples mount the host checkpoint directory at `/models/GLM-5.3-Flash`. Set + `MODEL_PATH=/models/GLM-5.3-Flash` in the launch scripts when using this mount. +- Give each container its own `NAME`. For PD disaggregation, start a container on both nodes. +- The examples use the SGLang package installed in the image and require no source-code path or `PYTHONPATH` override. + + +## Online service deployment + +### Single-node deployment + +Prefill and decode run in the same instance. Select the script for your platform and run it inside the container. + + + + +Use the ModelSlim W8A8 checkpoint on one Atlas 800I A3 node, with TP16. +Replace `MODEL_PATH` with the checkpoint directory inside the container. + +```shell +# ============================================================ +# Before running, update the following variables: +# MODEL_PATH: path to the model weights directory +# ============================================================ + +MODEL_PATH=/path/to/model-weights + +source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash +source /usr/local/Ascend/nnal/atb/set_env.sh + +export HCCL_BUFFSIZE=1536 +unset ASCEND_LAUNCH_BLOCKING + +python3 -m sglang.launch_server \ + --model-path "$MODEL_PATH" \ + --quantization modelslim \ + --attention-backend ascend \ + --device npu \ + --tp-size 16 --nnodes 1 \ + --chunked-prefill-size 4096 \ + --max-prefill-tokens 8192 \ + --trust-remote-code \ + --mem-fraction-static 0.78 \ + --page-size 128 \ + --served-model-name GLM-5.3-Flash \ + --cuda-graph-bs-decode 16 \ + --max-running-requests 16 \ + --load-balance-method round_robin \ + --enable-multimodal \ + --moe-a2a-backend deepep --deepep-mode auto \ + --speculative-draft-model-path "$MODEL_PATH" \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --watchdog-timeout 9000 \ + --host 127.0.0.1 --port 8810 +``` + + + + +Use the ModelSlim W8A8_MXFP8 checkpoint on one A5 node, with TP8 and no DP attention. +Replace `MODEL_PATH` with the checkpoint directory inside the container. + +```shell +# ============================================================ +# Before running, update the following variables: +# MODEL_PATH: path to the model weights directory +# ============================================================ + +MODEL_PATH=/path/to/model-weights + +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/memfabric_hybrid/set_env.sh + +export SGLANG_NPU_PROFILING=0 +export ASCEND_LAUNCH_BLOCKING=1 +export HCCL_BUFFSIZE=256 + +python3 -m sglang.launch_server \ + --model-path "$MODEL_PATH" \ + --quantization modelslim \ + --attention-backend ascend \ + --device npu \ + --tp-size 8 --nnodes 1 \ + --chunked-prefill-size 8192 \ + --max-prefill-tokens 8192 \ + --trust-remote-code \ + --mem-fraction-static 0.83 \ + --page-size 64 \ + --served-model-name GLM-5.3-Flash \ + --load-format auto \ + --max-running-requests 16 \ + --pre-warm-nccl \ + --speculative-draft-model-path "$MODEL_PATH" \ + --speculative-draft-kv-cache-dtype bf16 \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 4 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 5 \ + --watchdog-timeout 1200 \ + --cuda-graph-bs-decode 1 8 64 \ + --host 127.0.0.1 --port 8810 +``` + + + + +### Prefill-decode disaggregation deployment + +PD disaggregation places prefill and decode on separate nodes. Replace the model path, node IP addresses, and +network interface names, then run the same script on both nodes. The script selects its role by matching the local +IP address. + +Ensure the worker HTTP ports, prefill bootstrap port, and MemFabric store port are reachable between the nodes. + + + + +Use W8A8 weights on two Atlas 800I A3 nodes. The prefill instance uses TP4 / PP4 / DP4; the decode instance uses +TP16 / PP1 / DP16 and NEXTN speculative decoding with three steps and four draft tokens. +Run the CPU tuning commands with the required host privileges. + +```shell +# ============================================================ +# Before running, update the following variables: +# MODEL_PATH: path to the model weights directory +# PREFILL_IP: prefill node IP address +# DECODE_IP: decode node IP address +# HCCL_SOCKET_IFNAME: network interface name for HCCL +# GLOO_SOCKET_IFNAME: network interface name for Gloo +# ============================================================ + +MODEL_PATH=/path/to/model-weights +PREFILL_IP="" +DECODE_IP="" + +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +source /usr/local/Ascend/nnal/atb/set_env.sh + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export HCCL_SOCKET_IFNAME="" +export GLOO_SOCKET_IFNAME="" +export ASCEND_MF_STORE_URL="tcp://${PREFILL_IP}:24707" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 +export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 + +unset ASCEND_LAUNCH_BLOCKING +unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY +export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${no_proxy:-}" +export NO_PROXY="$no_proxy" +LOCAL_IPS=" $(hostname -I) " + +if [[ "$LOCAL_IPS" == *" $PREFILL_IP "* ]]; then + export SGLANG_HOST_IP="$PREFILL_IP" + export HCCL_BUFFSIZE=1200 + export DEEPEP_HCCL_BUFFSIZE=1200 + export DEEPEP_NORMAL_LONG_SEQ_ROUND=128 + export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 + export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + export TASK_QUEUE_ENABLE=2 + export HCCL_IF_BASE_PORT=61000 + + python3 -m sglang.launch_server \ + --model-path "$MODEL_PATH" \ + --quantization modelslim \ + --device npu \ + --attention-backend ascend \ + --trust-remote-code \ + --enable-multimodal \ + --served-model-name GLM-5.3-Flash \ + --disaggregation-mode prefill \ + --disaggregation-bootstrap-port 8998 \ + --disaggregation-transfer-backend ascend \ + --dist-init-addr "${PREFILL_IP}:5000" \ + --tp-size 4 --pp-size 4 --dp-size 4 --nnodes 1 \ + --enable-dp-attention \ + --enable-dp-lm-head \ + --moe-dense-tp 1 \ + --moe-a2a-backend deepep --deepep-mode normal \ + --load-balance-method round_robin \ + --mem-fraction-static 0.78 \ + --page-size 128 \ + --max-running-requests 16 \ + --chunked-prefill-size 4096 \ + --max-prefill-tokens 8192 \ + --watchdog-timeout 9000 \ + --host "$PREFILL_IP" --port 8810 +elif [[ "$LOCAL_IPS" == *" $DECODE_IP "* ]]; then + export SGLANG_HOST_IP="$DECODE_IP" + export HCCL_BUFFSIZE=512 + export DEEPEP_HCCL_BUFFSIZE=512 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 + export SGLANG_ENABLE_SPEC_V2=1 + export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 + export SGLANG_NPU_USE_MULTI_STREAM=1 + export TASK_QUEUE_ENABLE=0 + export HCCL_IF_BASE_PORT=61010 + + python3 -m sglang.launch_server \ + --model-path "$MODEL_PATH" \ + --quantization modelslim \ + --device npu \ + --attention-backend ascend \ + --trust-remote-code \ + --enable-multimodal \ + --served-model-name GLM-5.3-Flash \ + --disaggregation-mode decode \ + --disaggregation-transfer-backend ascend \ + --dist-init-addr "${DECODE_IP}:5000" \ + --tp-size 16 --pp-size 1 --dp-size 16 --nnodes 1 \ + --enable-dp-attention \ + --moe-a2a-backend deepep --deepep-mode low_latency \ + --load-balance-method round_robin \ + --mem-fraction-static 0.75 \ + --page-size 128 \ + --max-running-requests 32 \ + --cuda-graph-bs-decode 1 2 \ + --speculative-draft-model-path "$MODEL_PATH" \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --watchdog-timeout 9000 \ + --host "$DECODE_IP" --port 8811 +else + echo "Local IP does not match PREFILL_IP or DECODE_IP: $LOCAL_IPS" >&2 + exit 1 +fi +``` + + + + +Use W8A8_MXFP8 weights on two A5 nodes. Both instances use TP8 / PP1 / DP2 and NEXTN speculative decoding +with four steps and five draft tokens. The nodes must be able to communicate over the `device_urma` transport. + +```shell +# ============================================================ +# Before running, update the following variables: +# MODEL_PATH: path to the model weights directory +# PREFILL_IP: prefill node IP address +# DECODE_IP: decode node IP address +# ============================================================ + +MODEL_PATH=/path/to/model-weights +PREFILL_IP="" +DECODE_IP="" + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash + +export SGLANG_NPU_PROFILING=0 +export HCCL_BUFFSIZE=256 +export ASCEND_MF_STORE_URL="tcp://${PREFILL_IP}:24670" +export ASCEND_MF_TRANSFER_PROTOCOL=device_urma +unset ASCEND_LAUNCH_BLOCKING +export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${no_proxy:-}" +export NO_PROXY="$no_proxy" +LOCAL_IPS=" $(hostname -I) " + +if [[ "$LOCAL_IPS" == *" $PREFILL_IP "* ]]; then + export SGLANG_HOST_IP="$PREFILL_IP" + python3 -m sglang.launch_server \ + --model-path "$MODEL_PATH" \ + --quantization modelslim --load-format auto \ + --device npu --attention-backend ascend --trust-remote-code \ + --served-model-name GLM-5.3-Flash \ + --disaggregation-mode prefill \ + --disaggregation-bootstrap-port 8998 \ + --disaggregation-transfer-backend ascend \ + --tp-size 8 --pp-size 1 --dp-size 2 --enable-dp-attention --nnodes 1 \ + --chunked-prefill-size 16384 --max-prefill-tokens 16384 \ + --mem-fraction-static 0.83 --page-size 64 --max-running-requests 16 \ + --speculative-draft-model-path "$MODEL_PATH" \ + --speculative-draft-kv-cache-dtype bf16 \ + --speculative-algorithm NEXTN --speculative-num-steps 4 \ + --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --pre-warm-nccl --watchdog-timeout 1200 --cuda-graph-bs-decode 1 8 64 \ + --host "$PREFILL_IP" --port 8810 +elif [[ "$LOCAL_IPS" == *" $DECODE_IP "* ]]; then + export SGLANG_HOST_IP="$DECODE_IP" + python3 -m sglang.launch_server \ + --model-path "$MODEL_PATH" \ + --quantization modelslim --load-format auto \ + --device npu --attention-backend ascend --trust-remote-code \ + --served-model-name GLM-5.3-Flash \ + --disaggregation-mode decode \ + --disaggregation-bootstrap-port 8998 \ + --disaggregation-transfer-backend ascend \ + --tp-size 8 --pp-size 1 --dp-size 2 --enable-dp-attention --nnodes 1 \ + --chunked-prefill-size 16384 --max-prefill-tokens 16384 \ + --mem-fraction-static 0.83 --page-size 64 --max-running-requests 16 \ + --speculative-draft-model-path "$MODEL_PATH" \ + --speculative-draft-kv-cache-dtype bf16 \ + --speculative-algorithm NEXTN --speculative-num-steps 4 \ + --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --pre-warm-nccl --watchdog-timeout 1200 --cuda-graph-bs-decode 1 8 64 \ + --host "$DECODE_IP" --port 8811 +else + echo "Local IP does not match PREFILL_IP or DECODE_IP: $LOCAL_IPS" >&2 + exit 1 +fi +``` + + + + +Launch the router after the prefill and decode services are ready. Use the same node IP addresses as in the worker +scripts. Client requests go to the router on port 8000; the worker ports are 8810 and 8811, and the prefill bootstrap +port is 8998. + +```shell +# ============================================================ +# Before running, update the following variables: +# PREFILL_IP: prefill node IP address +# DECODE_IP: decode node IP address +# ROUTER_HOST_IP: router node IP address +# ============================================================ + +PREFILL_IP="" +DECODE_IP="" +ROUTER_HOST_IP="" +export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${ROUTER_HOST_IP},${no_proxy:-}" +export NO_PROXY="$no_proxy" + +python3 -m sglang_router.launch_router \ + --pd-disaggregation \ + --prefill "http://${PREFILL_IP}:8810" 8998 \ + --decode "http://${DECODE_IP}:8811" \ + --host "$ROUTER_HOST_IP" --port 8000 \ + --policy round_robin \ + --worker-startup-timeout-secs 3600 \ + --request-timeout-secs 7200 \ + --disable-retries +``` + +## Functional verification + +After the service starts, send a prompt to verify inference. For mixed deployment, run the command inside the +container with `HOST=127.0.0.1` and `PORT=8810`. For PD disaggregation, use the router IP address and `PORT=8000`. + +```shell +# ============================================================ +# Before running, update the following variables: +# HOST: server or router host address +# PORT: 8810 for mixed deployment; 8000 for the router +# ============================================================ + +HOST=127.0.0.1 +PORT=8810 + +curl "http://${HOST}:${PORT}/generate" \ + -H "Content-Type: application/json" \ + -d '{ + "text": "What is the capital of France?", + "sampling_params": { + "max_new_tokens": 64, + "temperature": 0 + } + }' +``` + +Check for an HTTP 200 response with the generated answer. Once the server prints +`The server is fired up and ready to roll!`, it is ready to accept requests. +For Health Check, Generate, Chat Completions, and port usage examples, see +[Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). + +## Accuracy evaluation + +For evaluation methods and datasets, see +[Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation). + +## Performance + +For benchmark methods and metric definitions, see +[Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing). + +For a generated shared-prefix benchmark workload, use approximately 128K input tokens and 1K output tokens. +For mixed deployment, use the worker host and port 8810. For PD disaggregation, use the router host and port 8000. +Set both `--model` and `--tokenizer` to the checkpoint path and use `--served-model-name GLM-5.3-Flash`. + +## Best practices + +### Best practice configuration reference + +The [Online service deployment](#online-service-deployment) section above contains complete scripts for +single-node mixed deployment and PD disaggregation. + +For the optimization feature catalog and compatibility details, see +[Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization). + +## Performance tuning + +For the feature overview, see [Supported features](#supported-features). For tuning parameters and feature +compatibility, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization). + +## FAQ + +For environment, installation, and general parameter issues, see +[Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/ascend_npu_faq).