[NPU] [DOC] Fix and update Ascend NPU docs (#29501)

This commit is contained in:
amote-i
2026-06-27 18:23:11 +08:00
committed by GitHub
parent e0c0c0a45c
commit a3c5e286f6
9 changed files with 45 additions and 23 deletions
@@ -1,7 +1,8 @@
--- ---
title: SGLang installation with NPUs support title: SGLang installation with NPUs support
description: "Complete installation guide for SGLang on Ascend NPUs, including component version mapping, environment setup, and launching inference services."
--- ---
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems. You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
## Component Version Mapping For SGLang ## Component Version Mapping For SGLang
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}> <table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
@@ -452,7 +453,7 @@ python3 -m sglang_router.launch_router \
#### PD Mixed Scene #### PD Mixed Scene
```bash Command ```bash Command
python3 -m sglang.launch_server \ python3 -m sglang.launch_server \
--model-path Qwen3-VL-30B-A3B-Instruct \ --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \
--host 127.0.0.1 \ --host 127.0.0.1 \
--port 8000 \ --port 8000 \
--tp 4 \ --tp 4 \
@@ -78,10 +78,10 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
### Install EvalScope ### Install EvalScope
```shell Command ```shell Command
# Method1 Installing via pip # Method 1: Installing via pip
pip install evalscope pip install evalscope
# Method2 Installing from source # Method 2: Installing from source
git clone https://github.com/modelscope/evalscope.git git clone https://github.com/modelscope/evalscope.git
cd evalscope/ cd evalscope/
pip install -e . pip install -e .
@@ -241,7 +241,27 @@ When using EvalScope commands without specifying a dataset or model path, it wil
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/gsm8k (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)'))) requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/gsm8k (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)')))
[ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception [ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
``` ```
You can navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify` to `False` to resolve this. **Temporary workaround (test only):**
Navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify = False`.
<Warning>
This **disables TLS certificate validation globally** for the Python `requests` library. Use it **only as a temporary diagnostic step** in isolated test environments — never in production.
</Warning>
**Stable solution:**
The error is caused by a corporate TLS proxy injecting a self-signed certificate. Point `requests` to the proxy's CA bundle:
```shell
# Obtain the CA certificate from your network administrator
# Then set the environment variable:
export REQUESTS_CA_BUNDLE=/path/to/your-proxy-ca-bundle.crt
```
<Note>
This is a common workaround for corporate proxy environments. If it does not resolve your issue, consult your IT department — proxy configurations vary across organizations.
</Note>
If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below.
### EvalScope Request Retry Timeout ### EvalScope Request Retry Timeout
If EvalScope keeps retrying requests with errors like: If EvalScope keeps retrying requests with errors like:
@@ -267,9 +287,9 @@ root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/
Connecting to 141.5.152.215:6688... connected. Connecting to 141.5.152.215:6688... connected.
ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘CN=Huawei Web Secure Internet Gateway CA V2,OU=IT,O=Huawei,L=Shenzhen,ST=GuangDong,C=CN’: ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘CN=Huawei Web Secure Internet Gateway CA V2,OU=IT,O=Huawei,L=Shenzhen,ST=GuangDong,C=CN’:
Self-signed certificate encountered. Self-signed certificate encountered.
To connect to www.modelscope.cn insecurely, use `--no-check-certificate'. To connect to www.modelscope.cn insecurely, use `--no-check-certificate`.
``` ```
You can add `--no-check-certificate' You can add `--no-check-certificate`
``` ```
wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate
``` ```
@@ -58,17 +58,17 @@ This document provides a list of commonly used environment variables and aims to
<tbody> <tbody>
<tr> <tr>
<td><code>DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS</code></td> <td><code>DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS</code></td>
<td>Enable ant-moving function in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank.</td> <td>Enable long-sequence token pipelining in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank.</td>
<td><code>8192</code></td> <td><code>8192</code></td>
</tr> </tr>
<tr> <tr>
<td><code>DEEPEP_NORMAL_LONG_SEQ_ROUND</code></td> <td><code>DEEPEP_NORMAL_LONG_SEQ_ROUND</code></td>
<td>Enable ant-moving function in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank.</td> <td>Enable long-sequence token pipelining in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank.</td>
<td><code>1</code></td> <td><code>1</code></td>
</tr> </tr>
<tr> <tr>
<td><code>DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ</code></td> <td><code>DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ</code></td>
<td>Enable ant-moving function in combine stage. <br/> The value <code>0</code> means disabled.</td> <td>Enable long-sequence token pipelining in combine stage. <br/> The value <code>0</code> means disabled.</td>
<td><code>0</code></td> <td><code>0</code></td>
</tr> </tr>
<tr> <tr>
@@ -138,7 +138,7 @@ The `alloc_extend_kernel` operator appears to have a memory allocation issue. Pe
Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead. Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead.
```text highlight=15-36 ```python highlight=15-36
def alloc_extend( def alloc_extend(
self, self,
prefix_lens: torch.Tensor, prefix_lens: torch.Tensor,
@@ -206,7 +206,7 @@ If the NPUs are not occupied, you can use `--tp` to deploy across multiple devic
## 5. How to update sgl-kernel-npu ## 5. How to update sgl-kernel-npu
### Solution ### Solution
``` ```shell
git clone https://github.com/sgl-project/sgl-kernel-npu.git git clone https://github.com/sgl-project/sgl-kernel-npu.git
source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/ascend-toolkit/set_env.sh
@@ -1,3 +1,8 @@
---
title: "GLM-5.2 examples"
description: "Deploy GLM-5.2 with SGLang on Ascend NPUs, including single-node, multi-node, and Prefill-Decode disaggregation modes."
---
# GLM-5.2 examples # GLM-5.2 examples
## Introduction ## Introduction
@@ -10,14 +15,14 @@ The GLM (General Language Model) series is an open-source bilingual large langua
- `GLM-5.2`(BF16 version): [Download model weight](https://huggingface.co/collections/zai-org/glm-52). - `GLM-5.2`(BF16 version): [Download model weight](https://huggingface.co/collections/zai-org/glm-52).
- `GLM-5.2-w8a8`(Quantized version without mtp): [Download model weight](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/). - `GLM-5.2-w8a8`(Quantized version without mtp): [Download model weight](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/).
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively. - You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively.
### Installation ### Installation
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
```{code-block} bash ```bash
#Atlas 800 A3 #Atlas 800 A3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.0.0-a3-glm5.2-20260615 docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.0.0-a3-glm5.2-20260615
#Atlas 800 A2 #Atlas 800 A2
@@ -250,7 +255,6 @@ do
break break
fi fi
done done
``` ```
### Prefill-Decode Disaggregation ### Prefill-Decode Disaggregation
@@ -356,9 +360,6 @@ do
break break
fi fi
done done
exit 1
``` ```
```shell ```shell
@@ -373,4 +374,4 @@ python3 -m sglang_router.launch_router \
### Using Benchmark ### Using Benchmark
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details. Refer to [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling) for details.
@@ -358,7 +358,7 @@ Max ITL (ms): 2229.30
#### SGLang Serving Benchmark Result — Complete Reference #### SGLang Serving Benchmark Result — Complete Reference
The output format is **hardcoded in `bench_serving.py`. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments. The output format is **hardcoded in `bench_serving.py`**. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments.
##### Test Configuration ##### Test Configuration
@@ -152,7 +152,7 @@ curl -X POST http://127.0.0.1:30000/stop_profile
server needs time to flush trace data to disk and parse it. Wait for the server needs time to flush trace data to disk and parse it. Wait for the
response to complete. response to complete.
This method takes a significant amount of time to parse This method takes a significant amount of time to parse
profiling data;consider using **Method B** instead to avoid lengthy waits. profiling data; consider using **Method B** instead to avoid lengthy waits.
</Note> </Note>
#### Method B: API Auto-Stop #### Method B: API Auto-Stop
@@ -5,7 +5,7 @@ metatags:
--- ---
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config. To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers. SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504) [ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
<table> <table>
@@ -1311,7 +1311,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
<tr> <tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--grammar-backend`</td> <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--grammar-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td> <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`xgrammar`</td> <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`xgrammar`,<br/>`outlines`,<br/>`llguidance`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td> <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr> </tr>
<tr> <tr>