minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU] fix ascend_docs (#28279)

Browse Source
This commit is contained in:
jianzhao-xu
2026-06-15 21:11:55 +08:00
committed by GitHub
parent 8bdb007e58
commit f8d1d397b6
4 changed files with 35 additions and 261 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
+4 -4
View File
@@ -127,9 +127,9 @@ pip install memfabric-hybrid==1.0.8
#### Pytorch and Pytorch Framework Adaptor on Ascend
```bash Command
PYTORCH_VERSION=2.8.0
TORCHVISION_VERSION=0.23.0
TORCH_NPU_VERSION=2.8.0.post2
PYTORCH_VERSION=2.10.0
TORCHVISION_VERSION=0.25.0
TORCH_NPU_VERSION=2.10.0
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
pip install torch_npu==$TORCH_NPU_VERSION
```
@@ -141,7 +141,7 @@ If you are using other versions of `torch` and install `torch_npu`, check [insta
We provide our own implementation of Triton for Ascend.
```bash Command
pip install triton-ascend
pip install triton-ascend==3.2.1
```
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md)
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx
+24 -250
View File
@@ -1,12 +1,12 @@
---
title: "Ascend NPU Accuracy Evaluation"
metatags:
description: "Complete guide for SGLang model accuracy evaluation on Ascend NPU using EvalScope and AISBench, covering text/multimodal scenarios."
description: "Complete guide for SGLang model accuracy evaluation on Ascend NPU using EvalScope, covering text/multimodal scenarios."
---
# Ascend NPU Accuracy Evaluation
This document describes how to perform accuracy evaluation for SGLang models running on Ascend NPU using two tools: **EvalScope** and **AISBench**. The following scenarios are covered:
This document describes how to perform accuracy evaluation for SGLang models running on Ascend NPU using a tool: **EvalScope**. The following scenarios are covered:
- **Online Testing**: Evaluate via API interface after starting SGLang server
- **Text Models**: Using Qwen2.5-7B-Instruct as example
@@ -87,6 +87,17 @@ cd evalscope/
pip install -e .
```
<Warning>
Before running any `evalscope` commands, make sure to unset proxy environment variables to prevent communication issues with the SGLang server:
```shell
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
```
</Warning>
### Online Text Model Testing
This section covers online evaluation scenarios where the SGLang server is already running.
@@ -111,7 +122,7 @@ evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
--eval-type server \
--eval-type openai_api \
--datasets gsm8k \
--limit 10
```
@@ -136,7 +147,7 @@ evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
--eval-type server \
--eval-type openai_api \
--datasets mmlu
# CEval (Chinese evaluation)
@@ -144,7 +155,7 @@ evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
--eval-type server \
--eval-type openai_api \
--datasets ceval
# MATH-500
@@ -152,15 +163,15 @@ evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
--eval-type server \
--datasets math
--eval-type openai_api \
--datasets math_500
# HumanEval (code generation)
evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
--eval-type server \
--eval-type openai_api \
--datasets humaneval
```
@@ -185,15 +196,15 @@ evalscope eval \
--model /home/weights/Qwen2.5-VL-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
--eval-type server \
--datasets mmbench
--eval-type openai_api \
--datasets mm_bench
# MMMU (multimodal comprehensive understanding)
evalscope eval \
--model /home/weights/Qwen2.5-VL-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
--eval-type server \
--eval-type openai_api \
--datasets mmmu
# HallusionBench (hallucination evaluation)
@@ -201,251 +212,14 @@ evalscope eval \
--model /home/weights/Qwen2.5-VL-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
--eval-type server \
--datasets hallusionbench
--eval-type openai_api \
--datasets hallusion_bench
```
For more details, refer to the [EvalScope documentation](https://evalscope.readthedocs.io/).
---
## Using AISBench
[AISBench](https://github.com/AISBench/benchmark) is an official benchmark testing tool from Ascend, supporting accuracy and performance evaluation across multiple datasets.
### Install AISBench
```shell Command
# Install from source (recommended to use Gitee mirror)
git clone https://github.com/AISBench/benchmark.git
cd benchmark/
# Install core package (use Aliyun mirror if network access is restricted)
pip3 install -e ./ --use-pep517
# If dependency installation times out, use Aliyun mirror:
# pip3 install -r requirements/runtime.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
# Verify installation
ais_bench -h
```
> **Note**: When using `pip install -e` (development mode), the `ais_bench` command may not be in PATH. Use `python3 -m ais_bench.benchmark.cli.main` as an alternative.
### Configuration File Setup
Each model task, dataset task, and result presentation task corresponds to a configuration file. You need to modify the content of these configuration files before running the command. The paths of these configuration files can be queried by adding `--search` to the original AISBench command. For example:
```
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds --search
```
Executing the query command will yield the following results:
```
╒═════════════╤══════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│ Task Type │ Task Name │ Config File Path │
╞═════════════╪══════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════╡
│ --models │ vllm_api_general_chat │ /home/code/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py │
├─────────────┼──────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ --datasets │ gsm8k_gen_0_shot_cot_chat_prompt │ /home/code/benchmark/ais_bench/benchmark/configs/datasets/gsm8k/gsm8k_gen_0_shot_cot_chat_prompt.py │
╘═════════════╧══════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════╛
```
For online text models, edit `benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`:
```python
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content
models = [
dict(
attr="service", # Backend type identifier
type=VLLMCustomAPIChat,
abbr='vllm-api-general-chat',
path="/home/weights/Qwen2.5-7B-Instruct", # Path to model vocabulary file (usually not required for accuracy testing)
model="/home/weights/Qwen2.5-7B-Instruct", # Model name on server (empty string auto-detects)
request_rate=0, # Request frequency; sends all at once if <0.1
retry=2, # Maximum retry attempts per request
host_ip="localhost", # Inference service IP
host_port=30000, # Inference service port
max_out_len=512, # Maximum output tokens
batch_size=1, # Maximum request concurrency
trust_remote_code=False, # Whether tokenizer trusts remote code
generation_kwargs=dict( # Inference parameters (passed directly to requests)
temperature=0.6,
top_k=10,
top_p=0.95,
seed=None,
repetition_penalty=1.03,
),
pred_postprocessor=dict(type=extract_non_reasoning_content)
)
]
```
> **Note**: SGLang server defaults to port `30000` and is compatible with OpenAI API format, so AISBench's `VLLMCustomAPIChat` can connect directly to SGLang.
>
> **Important**: The sum of `max_out_len` and input token count must not exceed the SGLang server's `max_model_len` (default 32768 for Qwen2.5-7B). We recommend setting `max_out_len` to `512` or `1024` to avoid `400` errors caused by exceeding the context window.
### Download Datasets
AISBench supports multiple common datasets that must be downloaded to a specified path before use.
```shell Command
# C-Eval
cd ais_bench/datasets
mkdir ceval/ && mkdir ceval/formal_ceval
cd ceval/formal_ceval
wget https://www.modelscope.cn/datasets/opencompass/ceval-exam/resolve/master/ceval-exam.zip
unzip ceval-exam.zip && rm ceval-exam.zip
cd ../../..
# MMLU
cd ais_bench/datasets
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/mmlu.zip
unzip mmlu.zip && rm mmlu.zip
cd ../..
# GSM8K
cd ais_bench/datasets
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gsm8k.zip
unzip gsm8k.zip && rm gsm8k.zip
cd ../..
# GPQA
cd ais_bench/datasets
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gpqa.zip
unzip gpqa.zip && rm gpqa.zip
cd ../..
# MATH-500
cd ais_bench/datasets
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/math.zip
unzip math.zip && rm math.zip
cd ../..
# AIME 2024
cd ais_bench/datasets
mkdir aime/ && cd aime/
wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/aime.zip
unzip aime.zip && rm aime.zip
cd ../../..
# MMStar
cd ais_bench/datasets
mkdir mmstar
cd mmstar
wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
cd ../..
# MMMU
cd ais_bench/datasets
git lfs install
git clone https://www.modelscope.cn/datasets/AI-ModelScope/MMMU.git mmmu
cd ../..
```
### Online Text Model Testing
#### Start SGLang Server
```shell Command
sglang serve --model-path /home/weights/Qwen2.5-7B-Instruct \
--attention-backend ascend \
--host 0.0.0.0 --port 30000 &
```
#### Execute Accuracy Evaluation
```shell Command
# Run C-Eval dataset
ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
# Run MMLU dataset
ais_bench --models vllm_api_general_chat --datasets mmlu_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
# Run GSM8K dataset
ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
# Run GPQA dataset
ais_bench --models vllm_api_general_chat --datasets gpqa_gen_0_shot_str.py --mode all --dump-eval-details --merge-ds
# Run MATH-500 dataset
ais_bench --models vllm_api_general_chat --datasets math500_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
# Run AIME 2024 dataset
ais_bench --models vllm_api_general_chat --datasets aime2024_gen_0_shot_chat_prompt.py --mode all --dump-eval-details --merge-ds
```
After execution, results are saved in `outputs/default/<timestamp>/` with the following structure:
```
outputs/default/20250628_151326/
├── configs # Configuration files
├── logs # Execution logs
│ ├── eval # Accuracy evaluation logs
│ └── infer # Inference process logs
├── predictions # Inference results (JSON)
├── results # Raw accuracy scores (JSON)
└── summary # Final result summary
├── summary_20250628_151326.csv
├── summary_20250628_151326.md
└── summary_20250628_151326.txt
```
### Online Multimodal Model Testing
#### Configuration File
Edit multimodal model configuration file (e.g., `vllm_api_stream_chat_mutiturn.py`):
```python
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr="vllm-multiturn-api-chat-stream",
path="/home/weights/Qwen2.5-VL-7B-Instruct",
model="/home/weights/Qwen2.5-VL-7B-Instruct",
stream=True,
request_rate=0,
retry=2,
api_key="",
host_ip="localhost",
host_port=30000,
url="",
max_out_len=512,
batch_size=1,
trust_remote_code=False,
generation_kwargs=dict(
temperature=0.01,
ignore_eos=False,
),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]
```
#### Start Multimodal Server and Execute Evaluation
```shell Command
# Start multimodal server
sglang serve --model-path Qwen/Qwen2.5-VL-7B-Instruct \
--attention-backend ascend \
--mm-attention-backend ascend_attn \
--host 0.0.0.0 --port 30000 &
# Run MMStar dataset
ais_bench --models vllm_api_stream_chat_mutiturn --datasets mmstar_gen --mode all --dump-eval-details --merge-ds
# Run MMMU dataset
ais_bench --models vllm_api_stream_chat_mutiturn --datasets mmmu_gen --mode all --dump-eval-details --merge-ds
```
For more details, refer to the [AISBench documentation](https://yh-ais-bench-benchmark.readthedocs.io).
---
## Troubleshooting
### SGLang Server Startup Failure
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx
+5 -5
View File
@@ -175,11 +175,11 @@ Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the aff
# )
# else:
out_indices = torch.empty(
(extend_num_tokens,),
dtype=torch.int32,
device=self.device,
)
out_indices = torch.empty(
(extend_num_tokens,),
dtype=torch.int32,
device=self.device,
)
...
```
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx
+2 -2
View File
@@ -417,8 +417,8 @@ import sgl_kernel_npu
class TestHelloworld(unittest.TestCase):
def test_helloworld_basic(self):
x = torch.randn(1024, dtype=torch.bfloat16, device="npu")
y = torch.randn(1024, dtype=torch.bfloat16, device="npu")
x = torch.randn(2048, dtype=torch.float16, device="npu")
y = torch.randn(2048, dtype=torch.float16, device="npu")
z = torch.ops.npu.helloworld(x, y)
expected = x + y