minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU] [DOCS] Improved the usability of Ascend NPU documents (#25735)

Browse Source
This commit is contained in:
amote-i
2026-05-19 16:22:22 +08:00
committed by GitHub
parent 78cb38ed5e
commit 1f7bf155c3
8 changed files with 468 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
+222 -22
View File
@@ -53,19 +53,29 @@ You can install SGLang using any of the methods below. Please go through `System
### Obtain CANN Image ### Obtain CANN Image
<Warning>
Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space.
</Warning>
You can obtain the dependency of a specified version of CANN through an image. You can obtain the dependency of a specified version of CANN through an image.
<Note> <Tabs>
CANN images and SGLang images are hosted at different registry addresses. Make sure to pull them from the correct location. <Tab title="Atlas 800I A3">
</Note>
```bash Command ```bash Command
# for Atlas 800I A3 and Ubuntu OS
docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11 docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11
# for Atlas 800I A2 and Ubuntu OS ```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11 docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
``` ```
</Tab>
</Tabs>
## Preparing the Running Environment ## Preparing the Running Environment
### Method 1: Installing from source with prerequisites ### Method 1: Installing from source with prerequisites
@@ -142,14 +152,42 @@ pip install -e python[all_npu]
### Method 2: Using Docker Image ### Method 2: Using Docker Image
#### Obtain Image #### Obtain Image
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image. You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
<Warning>
Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
1. Download SGLang image 1. Download SGLang image
<Tip>
We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.10-npu.rc1-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann8.5.0-a3`) if you need the latest development changes.
</Tip>
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command ```bash Command
dockerhub: docker.io/lmsysorg/sglang:$tag # Stable release
# Main-based tag, change main to specific version like v0.5.6, docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
# you can get image for specific version
Atlas 800I A3 : {main}-cann8.5.0-a3 # Daily build
Atlas 800I A2: {main}-cann8.5.0-910b docker pull quay.io/ascend/sglang:main-cann8.5.0-a3
``` ```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
# Stable release
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
# Daily build
docker pull quay.io/ascend/sglang:main-cann8.5.0-910b
```
</Tab>
</Tabs>
2. Build an image based on Dockerfile 2. Build an image based on Dockerfile
```bash Command ```bash Command
# Clone the SGLang repository # Clone the SGLang repository
@@ -165,7 +203,8 @@ docker build --build-arg TARGETARCH=<arch_tag> -t <image_name> -f npu.Dockerfile
#### Create Docker #### Create Docker
__Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters. __Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.
__Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container. <Tabs>
<Tab title="Atlas 800I A3">
```bash Command ```bash Command
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
@@ -185,6 +224,32 @@ drun --env "HF_TOKEN=<secret>" \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
``` ```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci_manager --device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
# Add HF_TOKEN env for download model by SGLang.
drun --env "HF_TOKEN=<secret>" \
<image_name> \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```
</Tab>
</Tabs>
<Tip>
SGLang will serve on `http://127.0.0.1:30000` by default. You can change the host and port by `--host` and `--port` parameters.
</Tip>
## System Settings ## System Settings
### CPU performance power scheme ### CPU performance power scheme
@@ -221,20 +286,51 @@ cat /proc/sys/vm/swappiness # shows 10
```bash Command ```bash Command
# Enabling CPU Affinity # Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_SET_CPU_AFFINITY=1
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--attention-backend ascend \
--host 127.0.0.1 \
--port 8000
``` ```
#### PD Disaggregation Scene #### PD Disaggregation Scene
1. Launch Prefill Server 1. Launch Prefill Server
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command ```bash Command
# Enabling CPU Affinity # Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_SET_CPU_AFFINITY=1
# PIP: recommended to config first Prefill Server IP # PREFILL_IP: IP address of the first Prefill Server
# PORT: one free port # FREE_PORT: any available port
# all sglang servers need to be config the same PIP and PORT, # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PIP:PORT" export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode prefill \
--disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8995 \
--attention-backend ascend \
--device npu \
--base-gpu-id 0 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8000
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma" export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \ python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \ --model-path meta-llama/Llama-3.1-8B-Instruct \
@@ -249,13 +345,39 @@ python3 -m sglang.launch_server \
--port 8000 --port 8000
``` ```
</Tab>
</Tabs>
2. Launch Decode Server 2. Launch Decode Server
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command ```bash Command
# PIP: recommended to config first Prefill Server IP # PREFILL_IP: IP address of the first Prefill Server
# PORT: one free port # FREE_PORT: any available port
# all sglang servers need to be config the same PIP and PORT, # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PIP:PORT" export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode decode \
--disaggregation-transfer-backend ascend \
--attention-backend ascend \
--device npu \
--base-gpu-id 1 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8001
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma" export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \ python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \ --model-path meta-llama/Llama-3.1-8B-Instruct \
@@ -269,6 +391,9 @@ python3 -m sglang.launch_server \
--port 8001 --port 8001
``` ```
</Tab>
</Tabs>
3. Launch Router 3. Launch Router
```bash Command ```bash Command
python3 -m sglang_router.launch_router \ python3 -m sglang_router.launch_router \
@@ -296,3 +421,78 @@ python3 -m sglang.launch_server \
--enable-multimodal \ --enable-multimodal \
--sampling-backend ascend --sampling-backend ascend
``` ```
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests.
### Which port to send requests to
The port you use depends on your deployment mode:
| Scenario | Where to send requests |
|---|---|
| Non-PD (single server) | The server's `--port` (e.g., `8000` in the examples above) |
| Non-PD (multi-node) | The primary node's (`--node-rank 0`) `--port`; do **not** send requests to worker nodes |
| PD disaggregation | The router's `--port` (e.g., `6688` in the examples above); do **not** send requests directly to prefill or decode servers |
<Tip>
SGLang serves on port `30000` by default if `--port` is not specified. The examples in this guide use explicit ports for clarity.
If you are using PD disaggregation, replace `8000` with your router's port (e.g., `6688`) in the following examples.
</Tip>
### Health Check
```bash Command
curl http://127.0.0.1:8000/health
```
A successful response returns HTTP 200 with an empty body.
### Generate (Native Endpoint)
```bash Command
curl http://127.0.0.1:8000/generate \
-H "Content-Type: application/json" \
-d '{
"text": "What is the capital of France?",
"sampling_params": {"temperature": 0, "max_new_tokens": 128}
}'
```
The expected output should contain "Paris".
### Chat Completions (OpenAI-Compatible)
```bash Command
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "What is the capital of France?"}]
}'
```
The expected output should contain "Paris".
### Multimodal Chat Completions
<Note>
The image URL in the example below references an external resource (`raw.githubusercontent.com`). Make sure the server has internet access so the image can be downloaded at inference time. Alternatively, you can use a locally accessible URL or base64-encoded image data.
</Note>
```bash Command
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3-VL-30B-A3B-Instruct",
"messages": [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
{"type": "text", "text": "Describe this image."}
]
}]
}'
```
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx
+31 -3
View File
@@ -16,11 +16,17 @@ This document describes how to perform accuracy evaluation for SGLang models run
## Environment Setup ## Environment Setup
<Warning>
Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
First, launch the SGLang environment using the provided container image: First, launch the SGLang environment using the provided container image:
<Tabs>
<Tab title="Atlas 800I A3">
```shell Command ```shell Command
# Atlas 800I A3 environment export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-a3
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
@@ -39,7 +45,29 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
$IMAGE $IMAGE
``` ```
**For Atlas 800I A2 users**: Replace the image tag with `main-cann8.5.0-910b` and adjust device mappings from `/dev/davinci[8-15]` to `/dev/davinci[0-7]`. </Tab>
<Tab title="Atlas 800I A2">
```shell Command
export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin \
--volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule \
--volume ~/.cache/:/root/.cache/ \
--entrypoint=bash \
$IMAGE
```
</Tab>
</Tabs>
--- ---
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_deepseek_example.mdx
+9 -1
View File
@@ -6,6 +6,10 @@ metatags:
## Running DeepSeek-V3 ## Running DeepSeek-V3
<Warning>
If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3. ### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3.
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8). W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
@@ -271,7 +275,11 @@ python -m sglang_router.launch_router \
--mini-lb --mini-lb
``` ```
### test gsm8k ## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
### Test GSM8K
```python Test GSM8K ```python Test GSM8K
from types import SimpleNamespace from types import SimpleNamespace
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx
+60
View File
@@ -221,3 +221,63 @@ python -c "import sgl_kernel_npu; print(sgl_kernel_npu.__path__)"
rm -rf sgl-kernel-npu rm -rf sgl-kernel-npu
``` ```
## 6. `[Errno 101] Network is unreachable` when downloading HuggingFace datasets
### Error message
```text highlight=1-2
'[Errno 101] Network is unreachable' thrown while requesting HEAD https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
Retrying in 1s [Retry 1/5].
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 2353, in <module>
run_benchmark(args)
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 1848, in run_benchmark
input_requests = get_dataset(args, tokenizer, model_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/__init__.py", line 44, in get_dataset
return dataset.load(tokenizer=tokenizer, model_id=model_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 45, in load
return sample_random_requests(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 89, in sample_random_requests
dataset_path = download_and_cache_hf_file(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/utils.py", line 98, in download_and_cache_hf_file
return hf_hub_download(repo_id=repo_id, filename=filename, repo_type=repo_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
RuntimeError: Cannot send a request, as the client has been closed.
[ERROR] 2026-05-18-11:58:31 (PID:215, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```
### Cause
The machine cannot directly access the HuggingFace server due to network restrictions (firewall, proxy, or regional access limitations).
### Solution
- **Use an HF mirror site** — set the `HF_ENDPOINT` environment variable to a mirror (e.g., hf-mirror.com):
```bash
export HF_ENDPOINT=https://hf-mirror.com
```
- **Use a proxy** — if you have an HTTP proxy available:
```bash
export http_proxy=http://your-proxy:port
export https_proxy=http://your-proxy:port
```
- **Download the dataset manually** — use a machine with network access to download the file, then transfer it to the target machine. Use `--dataset-path` to specify the local file path:
```bash
python -m sglang.bench_serving \
--dataset-name random \
--dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
...
```
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5_examples.mdx
+48 -7
View File
@@ -11,6 +11,10 @@ The GLM (General Language Model) series is an open-source bilingual large langua
### Model Weight ### Model Weight
<Warning>
If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
- `GLM-5.0`(BF16 version): [Download model weight](https://www.modelscope.cn/models/ZhipuAI/GLM-5). - `GLM-5.0`(BF16 version): [Download model weight](https://www.modelscope.cn/models/ZhipuAI/GLM-5).
- `GLM-5.0-w4a8`(Quantized version without mtp): [Download model weight](https://modelers.cn/models/Eco-Tech/GLM-5-w4a8). - `GLM-5.0-w4a8`(Quantized version without mtp): [Download model weight](https://modelers.cn/models/Eco-Tech/GLM-5-w4a8).
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively. - You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively.
@@ -18,15 +22,18 @@ The GLM (General Language Model) series is an open-source bilingual large langua
### Installation ### Installation
<Warning>
Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space.
</Warning>
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command ```bash Command
#Atlas 800 A3 docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-glm5
#Atlas 800 A2
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-glm5
#start container
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \ --privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \ -v /var/queue_schedule:/var/queue_schedule \
@@ -53,9 +60,39 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \ --device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \ --entrypoint=bash \
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG} quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
``` ```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
```
</Tab>
</Tabs>
### Best Practices ### Best Practices
Note: Using this image for **best practices**, you need to update transformers to version 5.3.0 Note: Using this image for **best practices**, you need to update transformers to version 5.3.0
``` ```
@@ -195,7 +232,11 @@ done
### Prefill-Decode Disaggregation ### Prefill-Decode Disaggregation
Not test yet. Not tested yet.
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
### Using Benchmark ### Using Benchmark
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx
+42 -5
View File
@@ -13,14 +13,23 @@ metatags:
## Setup environment using container ## Setup environment using container
__Notice:__ The following commands are based on Atlas 800I A3 machines. If you are using Atlas 800I A2, some changes are needed. <Warning>
Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
- The image tag needs to be `main-cann8.5.0-a3` for Atlas 800I A3 and `main-cann8.5.0-910b` for Atlas 800I A2. <Tip>
- The device mapping in `docker run` command needs to be changed to `davinci[0-7]` for Atlas 800I A2. We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.10-npu.rc1-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann8.5.0-a3`) if you need the latest development changes.
</Tip>
<Tabs>
<Tab title="Atlas 800I A3">
```shell Command ```shell Command
# For Atlas 800I A3 # Stable release
export IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3 export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
# Daily build
export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-a3
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
@@ -39,6 +48,34 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
$IMAGE $IMAGE
``` ```
</Tab>
<Tab title="Atlas 800I A2">
```shell Command
# Stable release
export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
# Daily build
export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-910b
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin \
--volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule \
--volume ~/.cache/:/root/.cache/ \
--entrypoint=bash \
$IMAGE
```
</Tab>
</Tabs>
## Usage ## Usage
The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version. The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version.
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx
+48 -11
View File
@@ -7,15 +7,18 @@ metatags:
### Installation ### Installation
<Warning>
Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command ```bash Command
#Atlas 800 A3 docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3
#Atlas 800 A2
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-910b
#start container
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \ --privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \ -v /var/queue_schedule:/var/queue_schedule \
@@ -42,9 +45,39 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \ --device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \ --entrypoint=bash \
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${tag} quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
``` ```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
```
</Tab>
</Tabs>
## Deployment ## Deployment
### Single-node Deployment ### Single-node Deployment
@@ -54,7 +87,7 @@ Run the following script to execute online inference.
#### Qwen3.5 397B #### Qwen3.5 397B
<Tip> <Tip>
Recommended model: `Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp` Recommended model: [`Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp)
</Tip> </Tip>
```bash Command ```bash Command
@@ -101,7 +134,7 @@ python3 -m sglang.launch_server \
#### Qwen3.5 122B #### Qwen3.5 122B
<Tip> <Tip>
Recommended model: `Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp` Recommended model: [`Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp)
</Tip> </Tip>
```bash Command ```bash Command
@@ -148,7 +181,7 @@ python3 -m sglang.launch_server \
#### Qwen3.5 35B #### Qwen3.5 35B
<Tip> <Tip>
Recommended model: `Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp` Recommended model: [`Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp)
</Tip> </Tip>
```bash Command ```bash Command
@@ -195,7 +228,7 @@ python3 -m sglang.launch_server \
#### Qwen3.5 27B #### Qwen3.5 27B
<Tip> <Tip>
Recommended model: `Eco-Tech/Qwen3.5-27B-w8a8-mtp` Recommended model: [`Eco-Tech/Qwen3.5-27B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-27B-w8a8-mtp)
</Tip> </Tip>
```bash Command ```bash Command
@@ -240,7 +273,11 @@ python3 -m sglang.launch_server \
### Prefill-Decode Disaggregation ### Prefill-Decode Disaggregation
Not test yet. Not tested yet.
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, Multimodal Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
### Using Benchmark ### Using Benchmark
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_examples.mdx
+8
View File
@@ -5,6 +5,10 @@ metatags:
--- ---
## Qwen3 examples ## Qwen3 examples
<Warning>
If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
### Running Qwen3 ### Running Qwen3
#### Running Qwen3-32B on 1 x Atlas 800I A3. #### Running Qwen3-32B on 1 x Atlas 800I A3.
@@ -204,3 +208,7 @@ python -m sglang.launch_server \
--model-path Qwen/Qwen3-VL-8B-Instruct \ --model-path Qwen/Qwen3-VL-8B-Instruct \
--mem-fraction-static 0.8 --mem-fraction-static 0.8
``` ```
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).