[NPU] [DOC] Fix and update Ascend NPU docs (#29501)
This commit is contained in:
@@ -1,7 +1,8 @@
|
|||||||
---
|
---
|
||||||
title: SGLang installation with NPUs support
|
title: SGLang installation with NPUs support
|
||||||
|
description: "Complete installation guide for SGLang on Ascend NPUs, including component version mapping, environment setup, and launching inference services."
|
||||||
---
|
---
|
||||||
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
|
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
|
||||||
|
|
||||||
## Component Version Mapping For SGLang
|
## Component Version Mapping For SGLang
|
||||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||||
@@ -452,7 +453,7 @@ python3 -m sglang_router.launch_router \
|
|||||||
#### PD Mixed Scene
|
#### PD Mixed Scene
|
||||||
```bash Command
|
```bash Command
|
||||||
python3 -m sglang.launch_server \
|
python3 -m sglang.launch_server \
|
||||||
--model-path Qwen3-VL-30B-A3B-Instruct \
|
--model-path Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||||
--host 127.0.0.1 \
|
--host 127.0.0.1 \
|
||||||
--port 8000 \
|
--port 8000 \
|
||||||
--tp 4 \
|
--tp 4 \
|
||||||
|
|||||||
@@ -78,10 +78,10 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
|
|||||||
### Install EvalScope
|
### Install EvalScope
|
||||||
|
|
||||||
```shell Command
|
```shell Command
|
||||||
# Method1 Installing via pip
|
# Method 1: Installing via pip
|
||||||
pip install evalscope
|
pip install evalscope
|
||||||
|
|
||||||
# Method2 Installing from source
|
# Method 2: Installing from source
|
||||||
git clone https://github.com/modelscope/evalscope.git
|
git clone https://github.com/modelscope/evalscope.git
|
||||||
cd evalscope/
|
cd evalscope/
|
||||||
pip install -e .
|
pip install -e .
|
||||||
@@ -241,7 +241,27 @@ When using EvalScope commands without specifying a dataset or model path, it wil
|
|||||||
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/gsm8k (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)')))
|
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/gsm8k (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)')))
|
||||||
[ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
|
[ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
|
||||||
```
|
```
|
||||||
You can navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify` to `False` to resolve this.
|
**Temporary workaround (test only):**
|
||||||
|
Navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify = False`.
|
||||||
|
|
||||||
|
<Warning>
|
||||||
|
This **disables TLS certificate validation globally** for the Python `requests` library. Use it **only as a temporary diagnostic step** in isolated test environments — never in production.
|
||||||
|
</Warning>
|
||||||
|
|
||||||
|
**Stable solution:**
|
||||||
|
The error is caused by a corporate TLS proxy injecting a self-signed certificate. Point `requests` to the proxy's CA bundle:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
# Obtain the CA certificate from your network administrator
|
||||||
|
# Then set the environment variable:
|
||||||
|
export REQUESTS_CA_BUNDLE=/path/to/your-proxy-ca-bundle.crt
|
||||||
|
```
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
This is a common workaround for corporate proxy environments. If it does not resolve your issue, consult your IT department — proxy configurations vary across organizations.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below.
|
||||||
|
|
||||||
### EvalScope Request Retry Timeout
|
### EvalScope Request Retry Timeout
|
||||||
If EvalScope keeps retrying requests with errors like:
|
If EvalScope keeps retrying requests with errors like:
|
||||||
@@ -267,9 +287,9 @@ root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/
|
|||||||
Connecting to 141.5.152.215:6688... connected.
|
Connecting to 141.5.152.215:6688... connected.
|
||||||
ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘CN=Huawei Web Secure Internet Gateway CA V2,OU=IT,O=Huawei,L=Shenzhen,ST=GuangDong,C=CN’:
|
ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘CN=Huawei Web Secure Internet Gateway CA V2,OU=IT,O=Huawei,L=Shenzhen,ST=GuangDong,C=CN’:
|
||||||
Self-signed certificate encountered.
|
Self-signed certificate encountered.
|
||||||
To connect to www.modelscope.cn insecurely, use `--no-check-certificate'.
|
To connect to www.modelscope.cn insecurely, use `--no-check-certificate`.
|
||||||
```
|
```
|
||||||
You can add `--no-check-certificate'
|
You can add `--no-check-certificate`
|
||||||
```
|
```
|
||||||
wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate
|
wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -58,17 +58,17 @@ This document provides a list of commonly used environment variables and aims to
|
|||||||
<tbody>
|
<tbody>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS</code></td>
|
<td><code>DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS</code></td>
|
||||||
<td>Enable ant-moving function in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank.</td>
|
<td>Enable long-sequence token pipelining in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank.</td>
|
||||||
<td><code>8192</code></td>
|
<td><code>8192</code></td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>DEEPEP_NORMAL_LONG_SEQ_ROUND</code></td>
|
<td><code>DEEPEP_NORMAL_LONG_SEQ_ROUND</code></td>
|
||||||
<td>Enable ant-moving function in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank.</td>
|
<td>Enable long-sequence token pipelining in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank.</td>
|
||||||
<td><code>1</code></td>
|
<td><code>1</code></td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ</code></td>
|
<td><code>DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ</code></td>
|
||||||
<td>Enable ant-moving function in combine stage. <br/> The value <code>0</code> means disabled.</td>
|
<td>Enable long-sequence token pipelining in combine stage. <br/> The value <code>0</code> means disabled.</td>
|
||||||
<td><code>0</code></td>
|
<td><code>0</code></td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
|
|||||||
@@ -138,7 +138,7 @@ The `alloc_extend_kernel` operator appears to have a memory allocation issue. Pe
|
|||||||
|
|
||||||
Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead.
|
Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead.
|
||||||
|
|
||||||
```text highlight=15-36
|
```python highlight=15-36
|
||||||
def alloc_extend(
|
def alloc_extend(
|
||||||
self,
|
self,
|
||||||
prefix_lens: torch.Tensor,
|
prefix_lens: torch.Tensor,
|
||||||
@@ -206,7 +206,7 @@ If the NPUs are not occupied, you can use `--tp` to deploy across multiple devic
|
|||||||
## 5. How to update sgl-kernel-npu
|
## 5. How to update sgl-kernel-npu
|
||||||
|
|
||||||
### Solution
|
### Solution
|
||||||
```
|
```shell
|
||||||
git clone https://github.com/sgl-project/sgl-kernel-npu.git
|
git clone https://github.com/sgl-project/sgl-kernel-npu.git
|
||||||
|
|
||||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||||
|
|||||||
@@ -1,3 +1,8 @@
|
|||||||
|
---
|
||||||
|
title: "GLM-5.2 examples"
|
||||||
|
description: "Deploy GLM-5.2 with SGLang on Ascend NPUs, including single-node, multi-node, and Prefill-Decode disaggregation modes."
|
||||||
|
---
|
||||||
|
|
||||||
# GLM-5.2 examples
|
# GLM-5.2 examples
|
||||||
|
|
||||||
## Introduction
|
## Introduction
|
||||||
@@ -10,14 +15,14 @@ The GLM (General Language Model) series is an open-source bilingual large langua
|
|||||||
|
|
||||||
- `GLM-5.2`(BF16 version): [Download model weight](https://huggingface.co/collections/zai-org/glm-52).
|
- `GLM-5.2`(BF16 version): [Download model weight](https://huggingface.co/collections/zai-org/glm-52).
|
||||||
- `GLM-5.2-w8a8`(Quantized version without mtp): [Download model weight](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/).
|
- `GLM-5.2-w8a8`(Quantized version without mtp): [Download model weight](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/).
|
||||||
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively.
|
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively.
|
||||||
|
|
||||||
|
|
||||||
### Installation
|
### Installation
|
||||||
|
|
||||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
|
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
|
||||||
|
|
||||||
```{code-block} bash
|
```bash
|
||||||
#Atlas 800 A3
|
#Atlas 800 A3
|
||||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.0.0-a3-glm5.2-20260615
|
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.0.0-a3-glm5.2-20260615
|
||||||
#Atlas 800 A2
|
#Atlas 800 A2
|
||||||
@@ -250,7 +255,6 @@ do
|
|||||||
break
|
break
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Prefill-Decode Disaggregation
|
### Prefill-Decode Disaggregation
|
||||||
@@ -356,9 +360,6 @@ do
|
|||||||
break
|
break
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
|
|
||||||
exit 1
|
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
@@ -373,4 +374,4 @@ python3 -m sglang_router.launch_router \
|
|||||||
|
|
||||||
### Using Benchmark
|
### Using Benchmark
|
||||||
|
|
||||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details.
|
Refer to [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling) for details.
|
||||||
|
|||||||
@@ -358,7 +358,7 @@ Max ITL (ms): 2229.30
|
|||||||
|
|
||||||
#### SGLang Serving Benchmark Result — Complete Reference
|
#### SGLang Serving Benchmark Result — Complete Reference
|
||||||
|
|
||||||
The output format is **hardcoded in `bench_serving.py`. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments.
|
The output format is **hardcoded in `bench_serving.py`**. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments.
|
||||||
|
|
||||||
##### Test Configuration
|
##### Test Configuration
|
||||||
|
|
||||||
|
|||||||
@@ -152,7 +152,7 @@ curl -X POST http://127.0.0.1:30000/stop_profile
|
|||||||
server needs time to flush trace data to disk and parse it. Wait for the
|
server needs time to flush trace data to disk and parse it. Wait for the
|
||||||
response to complete.
|
response to complete.
|
||||||
This method takes a significant amount of time to parse
|
This method takes a significant amount of time to parse
|
||||||
profiling data;consider using **Method B** instead to avoid lengthy waits.
|
profiling data; consider using **Method B** instead to avoid lengthy waits.
|
||||||
</Note>
|
</Note>
|
||||||
|
|
||||||
#### Method B: API Auto-Stop
|
#### Method B: API Auto-Stop
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ metatags:
|
|||||||
---
|
---
|
||||||
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
|
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
|
||||||
|
|
||||||
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.
|
SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.
|
||||||
|
|
||||||
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
|
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
|
||||||
<table>
|
<table>
|
||||||
|
|||||||
@@ -1311,7 +1311,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
|
|||||||
<tr>
|
<tr>
|
||||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--grammar-backend`</td>
|
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--grammar-backend`</td>
|
||||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
|
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
|
||||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`xgrammar`</td>
|
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`xgrammar`,<br/>`outlines`,<br/>`llguidance`</td>
|
||||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
|
|||||||
Reference in New Issue
Block a user