From a3c5e286f625e993ef6bcf859824fa7fe62b7e5c Mon Sep 17 00:00:00 2001 From: amote-i <49533125+amote-i@users.noreply.github.com> Date: Sat, 27 Jun 2026 18:23:11 +0800 Subject: [PATCH] [NPU] [DOC] Fix and update Ascend NPU docs (#29501) --- .../ascend-npus/ascend_npu.mdx | 5 ++-- .../ascend_npu_accuracy_evaluation.mdx | 30 +++++++++++++++---- .../ascend_npu_environment_variables.mdx | 6 ++-- .../ascend-npus/ascend_npu_faq.mdx | 4 +-- .../ascend_npu_glm5.2_examples.mdx | 15 +++++----- .../ascend_npu_performance_testing.mdx | 2 +- .../ascend-npus/ascend_npu_profiling.mdx | 2 +- .../ascend-npus/ascend_npu_quantization.mdx | 2 +- .../ascend_npu_support_features.mdx | 2 +- 9 files changed, 45 insertions(+), 23 deletions(-) diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx index 395568d49..b4b5ef1b8 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx @@ -1,7 +1,8 @@ --- title: SGLang installation with NPUs support +description: "Complete installation guide for SGLang on Ascend NPUs, including component version mapping, environment setup, and launching inference services." --- -You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems. +You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems. ## Component Version Mapping For SGLang @@ -452,7 +453,7 @@ python3 -m sglang_router.launch_router \ #### PD Mixed Scene ```bash Command python3 -m sglang.launch_server \ - --model-path Qwen3-VL-30B-A3B-Instruct \ + --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \ --host 127.0.0.1 \ --port 8000 \ --tp 4 \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx index 110315a62..2004ae422 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx @@ -78,10 +78,10 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ ### Install EvalScope ```shell Command -# Method1 Installing via pip +# Method 1: Installing via pip pip install evalscope -# Method2 Installing from source +# Method 2: Installing from source git clone https://github.com/modelscope/evalscope.git cd evalscope/ pip install -e . @@ -241,7 +241,27 @@ When using EvalScope commands without specifying a dataset or model path, it wil requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/gsm8k (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)'))) [ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception ``` -You can navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify` to `False` to resolve this. +**Temporary workaround (test only):** +Navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify = False`. + + +This **disables TLS certificate validation globally** for the Python `requests` library. Use it **only as a temporary diagnostic step** in isolated test environments — never in production. + + +**Stable solution:** +The error is caused by a corporate TLS proxy injecting a self-signed certificate. Point `requests` to the proxy's CA bundle: + +```shell +# Obtain the CA certificate from your network administrator +# Then set the environment variable: +export REQUESTS_CA_BUNDLE=/path/to/your-proxy-ca-bundle.crt +``` + + +This is a common workaround for corporate proxy environments. If it does not resolve your issue, consult your IT department — proxy configurations vary across organizations. + + +If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below. ### EvalScope Request Retry Timeout If EvalScope keeps retrying requests with errors like: @@ -267,9 +287,9 @@ root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/ Connecting to 141.5.152.215:6688... connected. ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘CN=Huawei Web Secure Internet Gateway CA V2,OU=IT,O=Huawei,L=Shenzhen,ST=GuangDong,C=CN’: Self-signed certificate encountered. -To connect to www.modelscope.cn insecurely, use `--no-check-certificate'. +To connect to www.modelscope.cn insecurely, use `--no-check-certificate`. ``` -You can add `--no-check-certificate' +You can add `--no-check-certificate` ``` wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate ``` diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx index 4a88e7052..4f8723ef5 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx @@ -58,17 +58,17 @@ This document provides a list of commonly used environment variables and aims to - + - + - + diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx index c0fab4ab9..dd321d94b 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx @@ -138,7 +138,7 @@ The `alloc_extend_kernel` operator appears to have a memory allocation issue. Pe Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead. -```text highlight=15-36 +```python highlight=15-36 def alloc_extend( self, prefix_lens: torch.Tensor, @@ -206,7 +206,7 @@ If the NPUs are not occupied, you can use `--tp` to deploy across multiple devic ## 5. How to update sgl-kernel-npu ### Solution -``` +```shell git clone https://github.com/sgl-project/sgl-kernel-npu.git source /usr/local/Ascend/ascend-toolkit/set_env.sh diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5.2_examples.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5.2_examples.mdx index cb83c36bf..e560bc30d 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5.2_examples.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5.2_examples.mdx @@ -1,3 +1,8 @@ +--- +title: "GLM-5.2 examples" +description: "Deploy GLM-5.2 with SGLang on Ascend NPUs, including single-node, multi-node, and Prefill-Decode disaggregation modes." +--- + # GLM-5.2 examples ## Introduction @@ -10,14 +15,14 @@ The GLM (General Language Model) series is an open-source bilingual large langua - `GLM-5.2`(BF16 version): [Download model weight](https://huggingface.co/collections/zai-org/glm-52). - `GLM-5.2-w8a8`(Quantized version without mtp): [Download model weight](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/). -- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively. +- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively. ### Installation The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. -```{code-block} bash +```bash #Atlas 800 A3 docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.0.0-a3-glm5.2-20260615 #Atlas 800 A2 @@ -250,7 +255,6 @@ do break fi done - ``` ### Prefill-Decode Disaggregation @@ -356,9 +360,6 @@ do break fi done - -exit 1 - ``` ```shell @@ -373,4 +374,4 @@ python3 -m sglang_router.launch_router \ ### Using Benchmark -Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details. +Refer to [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling) for details. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx index 7035dbd74..de51acf2f 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx @@ -358,7 +358,7 @@ Max ITL (ms): 2229.30 #### SGLang Serving Benchmark Result — Complete Reference -The output format is **hardcoded in `bench_serving.py`. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments. +The output format is **hardcoded in `bench_serving.py`**. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments. ##### Test Configuration diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx index f9df1af39..e36747e36 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx @@ -152,7 +152,7 @@ curl -X POST http://127.0.0.1:30000/stop_profile server needs time to flush trace data to disk and parse it. Wait for the response to complete. This method takes a significant amount of time to parse -profiling data;consider using **Method B** instead to avoid lengthy waits. +profiling data; consider using **Method B** instead to avoid lengthy waits. #### Method B: API Auto-Stop diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx index c9fa5dd28..bbb7ced8f 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx @@ -5,7 +5,7 @@ metatags: --- To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config. -SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers. +SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers. [ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENSEnable ant-moving function in dispatch stage. Indicates
the number of tokens transmitted per round on each rank.		Enable long-sequence token pipelining in dispatch stage. Indicates
the number of tokens transmitted per round on each rank.		 8192
DEEPEP_NORMAL_LONG_SEQ_ROUNDEnable ant-moving function in dispatch stage. Indicates
the number of rounds transmitted on each rank.		Enable long-sequence token pipelining in dispatch stage. Indicates
the number of rounds transmitted on each rank.		 1
DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQEnable ant-moving function in combine stage.
The value 0 means disabled.		Enable long-sequence token pipelining in combine stage.
The value 0 means disabled.		 0
diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx index 24f592454..302bf6d84 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx @@ -1311,7 +1311,7 @@ click [Server Arguments](../../advanced_features/server_arguments). - +
`--grammar-backend` `None``xgrammar``xgrammar`,
`outlines`,
`llguidance`		 A2, A3