diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
index 1debe6d93..016a57e1e 100644
--- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
+++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
@@ -127,9 +127,9 @@ pip install memfabric-hybrid==1.0.8
#### Pytorch and Pytorch Framework Adaptor on Ascend
```bash Command
-PYTORCH_VERSION=2.8.0
-TORCHVISION_VERSION=0.23.0
-TORCH_NPU_VERSION=2.8.0.post2
+PYTORCH_VERSION=2.10.0
+TORCHVISION_VERSION=0.25.0
+TORCH_NPU_VERSION=2.10.0
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
pip install torch_npu==$TORCH_NPU_VERSION
```
@@ -141,7 +141,7 @@ If you are using other versions of `torch` and install `torch_npu`, check [insta
We provide our own implementation of Triton for Ascend.
```bash Command
-pip install triton-ascend
+pip install triton-ascend==3.2.1
```
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md)
diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx
index 7c5292bfd..d367d7dc2 100644
--- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx
+++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx
@@ -1,12 +1,12 @@
---
title: "Ascend NPU Accuracy Evaluation"
metatags:
- description: "Complete guide for SGLang model accuracy evaluation on Ascend NPU using EvalScope and AISBench, covering text/multimodal scenarios."
+ description: "Complete guide for SGLang model accuracy evaluation on Ascend NPU using EvalScope, covering text/multimodal scenarios."
---
# Ascend NPU Accuracy Evaluation
-This document describes how to perform accuracy evaluation for SGLang models running on Ascend NPU using two tools: **EvalScope** and **AISBench**. The following scenarios are covered:
+This document describes how to perform accuracy evaluation for SGLang models running on Ascend NPU using a tool: **EvalScope**. The following scenarios are covered:
- **Online Testing**: Evaluate via API interface after starting SGLang server
- **Text Models**: Using Qwen2.5-7B-Instruct as example
@@ -87,6 +87,17 @@ cd evalscope/
pip install -e .
```
+
+Before running any `evalscope` commands, make sure to unset proxy environment variables to prevent communication issues with the SGLang server:
+
+```shell
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+```
+
+
### Online Text Model Testing
This section covers online evaluation scenarios where the SGLang server is already running.
@@ -111,7 +122,7 @@ evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
- --eval-type server \
+ --eval-type openai_api \
--datasets gsm8k \
--limit 10
```
@@ -136,7 +147,7 @@ evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
- --eval-type server \
+ --eval-type openai_api \
--datasets mmlu
# CEval (Chinese evaluation)
@@ -144,7 +155,7 @@ evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
- --eval-type server \
+ --eval-type openai_api \
--datasets ceval
# MATH-500
@@ -152,15 +163,15 @@ evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
- --eval-type server \
- --datasets math
+ --eval-type openai_api \
+ --datasets math_500
# HumanEval (code generation)
evalscope eval \
--model /home/weights/Qwen2.5-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
- --eval-type server \
+ --eval-type openai_api \
--datasets humaneval
```
@@ -185,15 +196,15 @@ evalscope eval \
--model /home/weights/Qwen2.5-VL-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
- --eval-type server \
- --datasets mmbench
+ --eval-type openai_api \
+ --datasets mm_bench
# MMMU (multimodal comprehensive understanding)
evalscope eval \
--model /home/weights/Qwen2.5-VL-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
- --eval-type server \
+ --eval-type openai_api \
--datasets mmmu
# HallusionBench (hallucination evaluation)
@@ -201,251 +212,14 @@ evalscope eval \
--model /home/weights/Qwen2.5-VL-7B-Instruct \
--api-url http://localhost:30000/v1 \
--api-key EMPTY \
- --eval-type server \
- --datasets hallusionbench
+ --eval-type openai_api \
+ --datasets hallusion_bench
```
For more details, refer to the [EvalScope documentation](https://evalscope.readthedocs.io/).
---
-## Using AISBench
-
-[AISBench](https://github.com/AISBench/benchmark) is an official benchmark testing tool from Ascend, supporting accuracy and performance evaluation across multiple datasets.
-
-### Install AISBench
-
-```shell Command
-# Install from source (recommended to use Gitee mirror)
-git clone https://github.com/AISBench/benchmark.git
-cd benchmark/
-
-# Install core package (use Aliyun mirror if network access is restricted)
-pip3 install -e ./ --use-pep517
-
-# If dependency installation times out, use Aliyun mirror:
-# pip3 install -r requirements/runtime.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
-
-# Verify installation
-ais_bench -h
-```
-
-> **Note**: When using `pip install -e` (development mode), the `ais_bench` command may not be in PATH. Use `python3 -m ais_bench.benchmark.cli.main` as an alternative.
-
-### Configuration File Setup
-
-Each model task, dataset task, and result presentation task corresponds to a configuration file. You need to modify the content of these configuration files before running the command. The paths of these configuration files can be queried by adding `--search` to the original AISBench command. For example:
-```
-ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds --search
-```
-Executing the query command will yield the following results:
-```
-╒═════════════╤══════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════╕
-│ Task Type │ Task Name │ Config File Path │
-╞═════════════╪══════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════╡
-│ --models │ vllm_api_general_chat │ /home/code/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py │
-├─────────────┼──────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
-│ --datasets │ gsm8k_gen_0_shot_cot_chat_prompt │ /home/code/benchmark/ais_bench/benchmark/configs/datasets/gsm8k/gsm8k_gen_0_shot_cot_chat_prompt.py │
-╘═════════════╧══════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════╛
-```
-For online text models, edit `benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`:
-
-```python
-from ais_bench.benchmark.models import VLLMCustomAPIChat
-from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content
-
-models = [
- dict(
- attr="service", # Backend type identifier
- type=VLLMCustomAPIChat,
- abbr='vllm-api-general-chat',
- path="/home/weights/Qwen2.5-7B-Instruct", # Path to model vocabulary file (usually not required for accuracy testing)
- model="/home/weights/Qwen2.5-7B-Instruct", # Model name on server (empty string auto-detects)
- request_rate=0, # Request frequency; sends all at once if <0.1
- retry=2, # Maximum retry attempts per request
- host_ip="localhost", # Inference service IP
- host_port=30000, # Inference service port
- max_out_len=512, # Maximum output tokens
- batch_size=1, # Maximum request concurrency
- trust_remote_code=False, # Whether tokenizer trusts remote code
- generation_kwargs=dict( # Inference parameters (passed directly to requests)
- temperature=0.6,
- top_k=10,
- top_p=0.95,
- seed=None,
- repetition_penalty=1.03,
- ),
- pred_postprocessor=dict(type=extract_non_reasoning_content)
- )
-]
-```
-
-> **Note**: SGLang server defaults to port `30000` and is compatible with OpenAI API format, so AISBench's `VLLMCustomAPIChat` can connect directly to SGLang.
->
-> **Important**: The sum of `max_out_len` and input token count must not exceed the SGLang server's `max_model_len` (default 32768 for Qwen2.5-7B). We recommend setting `max_out_len` to `512` or `1024` to avoid `400` errors caused by exceeding the context window.
-
-### Download Datasets
-
-AISBench supports multiple common datasets that must be downloaded to a specified path before use.
-
-```shell Command
-# C-Eval
-cd ais_bench/datasets
-mkdir ceval/ && mkdir ceval/formal_ceval
-cd ceval/formal_ceval
-wget https://www.modelscope.cn/datasets/opencompass/ceval-exam/resolve/master/ceval-exam.zip
-unzip ceval-exam.zip && rm ceval-exam.zip
-cd ../../..
-
-# MMLU
-cd ais_bench/datasets
-wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/mmlu.zip
-unzip mmlu.zip && rm mmlu.zip
-cd ../..
-
-# GSM8K
-cd ais_bench/datasets
-wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gsm8k.zip
-unzip gsm8k.zip && rm gsm8k.zip
-cd ../..
-
-# GPQA
-cd ais_bench/datasets
-wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gpqa.zip
-unzip gpqa.zip && rm gpqa.zip
-cd ../..
-
-# MATH-500
-cd ais_bench/datasets
-wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/math.zip
-unzip math.zip && rm math.zip
-cd ../..
-
-# AIME 2024
-cd ais_bench/datasets
-mkdir aime/ && cd aime/
-wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/aime.zip
-unzip aime.zip && rm aime.zip
-cd ../../..
-
-# MMStar
-cd ais_bench/datasets
-mkdir mmstar
-cd mmstar
-wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
-cd ../..
-
-# MMMU
-cd ais_bench/datasets
-git lfs install
-git clone https://www.modelscope.cn/datasets/AI-ModelScope/MMMU.git mmmu
-cd ../..
-```
-
-### Online Text Model Testing
-
-#### Start SGLang Server
-
-```shell Command
-sglang serve --model-path /home/weights/Qwen2.5-7B-Instruct \
- --attention-backend ascend \
- --host 0.0.0.0 --port 30000 &
-```
-
-#### Execute Accuracy Evaluation
-
-```shell Command
-# Run C-Eval dataset
-ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
-
-# Run MMLU dataset
-ais_bench --models vllm_api_general_chat --datasets mmlu_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
-
-# Run GSM8K dataset
-ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
-
-# Run GPQA dataset
-ais_bench --models vllm_api_general_chat --datasets gpqa_gen_0_shot_str.py --mode all --dump-eval-details --merge-ds
-
-# Run MATH-500 dataset
-ais_bench --models vllm_api_general_chat --datasets math500_gen_0_shot_cot_chat_prompt.py --mode all --dump-eval-details --merge-ds
-
-# Run AIME 2024 dataset
-ais_bench --models vllm_api_general_chat --datasets aime2024_gen_0_shot_chat_prompt.py --mode all --dump-eval-details --merge-ds
-```
-
-After execution, results are saved in `outputs/default//` with the following structure:
-
-```
-outputs/default/20250628_151326/
-├── configs # Configuration files
-├── logs # Execution logs
-│ ├── eval # Accuracy evaluation logs
-│ └── infer # Inference process logs
-├── predictions # Inference results (JSON)
-├── results # Raw accuracy scores (JSON)
-└── summary # Final result summary
- ├── summary_20250628_151326.csv
- ├── summary_20250628_151326.md
- └── summary_20250628_151326.txt
-```
-
-### Online Multimodal Model Testing
-
-#### Configuration File
-
-Edit multimodal model configuration file (e.g., `vllm_api_stream_chat_mutiturn.py`):
-
-```python
-from ais_bench.benchmark.models import VLLMCustomAPIChat
-from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content
-
-models = [
- dict(
- attr="service",
- type=VLLMCustomAPIChat,
- abbr="vllm-multiturn-api-chat-stream",
- path="/home/weights/Qwen2.5-VL-7B-Instruct",
- model="/home/weights/Qwen2.5-VL-7B-Instruct",
- stream=True,
- request_rate=0,
- retry=2,
- api_key="",
- host_ip="localhost",
- host_port=30000,
- url="",
- max_out_len=512,
- batch_size=1,
- trust_remote_code=False,
- generation_kwargs=dict(
- temperature=0.01,
- ignore_eos=False,
- ),
- pred_postprocessor=dict(type=extract_non_reasoning_content),
- )
-]
-```
-
-#### Start Multimodal Server and Execute Evaluation
-
-```shell Command
-# Start multimodal server
-sglang serve --model-path Qwen/Qwen2.5-VL-7B-Instruct \
- --attention-backend ascend \
- --mm-attention-backend ascend_attn \
- --host 0.0.0.0 --port 30000 &
-
-# Run MMStar dataset
-ais_bench --models vllm_api_stream_chat_mutiturn --datasets mmstar_gen --mode all --dump-eval-details --merge-ds
-
-# Run MMMU dataset
-ais_bench --models vllm_api_stream_chat_mutiturn --datasets mmmu_gen --mode all --dump-eval-details --merge-ds
-```
-
-For more details, refer to the [AISBench documentation](https://yh-ais-bench-benchmark.readthedocs.io).
-
----
-
## Troubleshooting
### SGLang Server Startup Failure
diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx
index 78d3edae6..b16d1cefb 100644
--- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx
+++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx
@@ -175,11 +175,11 @@ Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the aff
# )
# else:
- out_indices = torch.empty(
- (extend_num_tokens,),
- dtype=torch.int32,
- device=self.device,
- )
+ out_indices = torch.empty(
+ (extend_num_tokens,),
+ dtype=torch.int32,
+ device=self.device,
+ )
...
```
diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx
index e1ab569b4..2488d64eb 100644
--- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx
+++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx
@@ -417,8 +417,8 @@ import sgl_kernel_npu
class TestHelloworld(unittest.TestCase):
def test_helloworld_basic(self):
- x = torch.randn(1024, dtype=torch.bfloat16, device="npu")
- y = torch.randn(1024, dtype=torch.bfloat16, device="npu")
+ x = torch.randn(2048, dtype=torch.float16, device="npu")
+ y = torch.randn(2048, dtype=torch.float16, device="npu")
z = torch.ops.npu.helloworld(x, y)
expected = x + y