minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[NPU] [DOC] Fix and update Ascend NPU docs (#29501)

Browse Source
This commit is contained in:
amote-i
2026-06-27 18:23:11 +08:00
committed by GitHub
parent e0c0c0a45c
commit a3c5e286f6
9 changed files with 45 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
+3 -2
View File
@@ -1,7 +1,8 @@
---
title: SGLang installation with NPUs support
description: "Complete installation guide for SGLang on Ascend NPUs, including component version mapping, environment setup, and launching inference services."
---
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
## Component Version Mapping For SGLang
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
@@ -452,7 +453,7 @@ python3 -m sglang_router.launch_router \
#### PD Mixed Scene
```bash Command
python3 -m sglang.launch_server \
--model-path Qwen3-VL-30B-A3B-Instruct \
--model-path Qwen/Qwen3-VL-30B-A3B-Instruct \
--host 127.0.0.1 \
--port 8000 \
--tp 4 \
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx
+25 -5
View File
@@ -78,10 +78,10 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
### Install EvalScope
```shell Command
# Method1 Installing via pip
# Method 1: Installing via pip
pip install evalscope
# Method2 Installing from source
# Method 2: Installing from source
git clone https://github.com/modelscope/evalscope.git
cd evalscope/
pip install -e .
@@ -241,7 +241,27 @@ When using EvalScope commands without specifying a dataset or model path, it wil
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/gsm8k (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)')))
[ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```
You can navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify` to `False` to resolve this.
**Temporary workaround (test only):**
Navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify = False`.
<Warning>
This **disables TLS certificate validation globally** for the Python `requests` library. Use it **only as a temporary diagnostic step** in isolated test environments — never in production.
</Warning>
**Stable solution:**
The error is caused by a corporate TLS proxy injecting a self-signed certificate. Point `requests` to the proxy's CA bundle:
```shell
# Obtain the CA certificate from your network administrator
# Then set the environment variable:
export REQUESTS_CA_BUNDLE=/path/to/your-proxy-ca-bundle.crt
```
<Note>
This is a common workaround for corporate proxy environments. If it does not resolve your issue, consult your IT department — proxy configurations vary across organizations.
</Note>
If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below.
### EvalScope Request Retry Timeout
If EvalScope keeps retrying requests with errors like:
@@ -267,9 +287,9 @@ root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/
Connecting to 141.5.152.215:6688... connected.
ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘CN=Huawei Web Secure Internet Gateway CA V2,OU=IT,O=Huawei,L=Shenzhen,ST=GuangDong,C=CN’:
Self-signed certificate encountered.
To connect to www.modelscope.cn insecurely, use `--no-check-certificate'.
To connect to www.modelscope.cn insecurely, use `--no-check-certificate`.
```
You can add `--no-check-certificate'
You can add `--no-check-certificate`
```
wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate
```
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx
+3 -3
View File
@@ -58,17 +58,17 @@ This document provides a list of commonly used environment variables and aims to
<tbody>
<tr>
<td><code>DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS</code></td>
<td>Enable ant-moving function in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank.</td>
<td>Enable long-sequence token pipelining in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank.</td>
<td><code>8192</code></td>
</tr>
<tr>
<td><code>DEEPEP_NORMAL_LONG_SEQ_ROUND</code></td>
<td>Enable ant-moving function in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank.</td>
<td>Enable long-sequence token pipelining in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank.</td>
<td><code>1</code></td>
</tr>
<tr>
<td><code>DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ</code></td>
<td>Enable ant-moving function in combine stage. <br/> The value <code>0</code> means disabled.</td>
<td>Enable long-sequence token pipelining in combine stage. <br/> The value <code>0</code> means disabled.</td>
<td><code>0</code></td>
</tr>
<tr>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx
+2 -2
View File
@@ -138,7 +138,7 @@ The `alloc_extend_kernel` operator appears to have a memory allocation issue. Pe
Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead.
```text highlight=15-36
```python highlight=15-36
def alloc_extend(
self,
prefix_lens: torch.Tensor,
@@ -206,7 +206,7 @@ If the NPUs are not occupied, you can use `--tp` to deploy across multiple devic
## 5. How to update sgl-kernel-npu
### Solution
```
```shell
git clone https://github.com/sgl-project/sgl-kernel-npu.git
source /usr/local/Ascend/ascend-toolkit/set_env.sh
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5.2_examples.mdx
+8 -7
View File
@@ -1,3 +1,8 @@
---
title: "GLM-5.2 examples"
description: "Deploy GLM-5.2 with SGLang on Ascend NPUs, including single-node, multi-node, and Prefill-Decode disaggregation modes."
---
# GLM-5.2 examples
## Introduction
@@ -10,14 +15,14 @@ The GLM (General Language Model) series is an open-source bilingual large langua
- `GLM-5.2`(BF16 version): [Download model weight](https://huggingface.co/collections/zai-org/glm-52).
- `GLM-5.2-w8a8`(Quantized version without mtp): [Download model weight](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/).
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively.
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively.
### Installation
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
```{code-block} bash
```bash
#Atlas 800 A3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.0.0-a3-glm5.2-20260615
#Atlas 800 A2
@@ -250,7 +255,6 @@ do
break
fi
done
```
### Prefill-Decode Disaggregation
@@ -356,9 +360,6 @@ do
break
fi
done
exit 1
```
```shell
@@ -373,4 +374,4 @@ python3 -m sglang_router.launch_router \
### Using Benchmark
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details.
Refer to [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling) for details.
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx
+1 -1
View File
@@ -358,7 +358,7 @@ Max ITL (ms): 2229.30
#### SGLang Serving Benchmark Result — Complete Reference
The output format is **hardcoded in `bench_serving.py`. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments.
The output format is **hardcoded in `bench_serving.py`**. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments.
##### Test Configuration
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx
+1 -1
View File
@@ -152,7 +152,7 @@ curl -X POST http://127.0.0.1:30000/stop_profile
server needs time to flush trace data to disk and parse it. Wait for the
response to complete.
This method takes a significant amount of time to parse
profiling data;consider using **Method B** instead to avoid lengthy waits.
profiling data; consider using **Method B** instead to avoid lengthy waits.
</Note>
#### Method B: API Auto-Stop
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx
+1 -1
View File
@@ -5,7 +5,7 @@ metatags:
---
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.
SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
<table>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx
+1 -1
View File
@@ -1311,7 +1311,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--grammar-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`xgrammar`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`xgrammar`,<br/>`outlines`,<br/>`llguidance`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>