minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[NPU] [DOCS] Improved the usability of Ascend NPU documents (#25735)

Browse Source
This commit is contained in:
amote-i
2026-05-19 16:22:22 +08:00
committed by GitHub
parent 78cb38ed5e
commit 1f7bf155c3
8 changed files with 468 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
+222 -22
View File
@@ -53,19 +53,29 @@ You can install SGLang using any of the methods below. Please go through `System
### Obtain CANN Image
<Warning>
Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space.
</Warning>
You can obtain the dependency of a specified version of CANN through an image.
<Note>
CANN images and SGLang images are hosted at different registry addresses. Make sure to pull them from the correct location.
</Note>
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command
# for Atlas 800I A3 and Ubuntu OS
docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11
# for Atlas 800I A2 and Ubuntu OS
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
```
</Tab>
</Tabs>
## Preparing the Running Environment
### Method 1: Installing from source with prerequisites
@@ -142,14 +152,42 @@ pip install -e python[all_npu]
### Method 2: Using Docker Image
#### Obtain Image
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
<Warning>
Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
1. Download SGLang image
<Tip>
We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.10-npu.rc1-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann8.5.0-a3`) if you need the latest development changes.
</Tip>
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command
dockerhub: docker.io/lmsysorg/sglang:$tag
# Main-based tag, change main to specific version like v0.5.6,
# you can get image for specific version
Atlas 800I A3 : {main}-cann8.5.0-a3
Atlas 800I A2: {main}-cann8.5.0-910b
# Stable release
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
# Daily build
docker pull quay.io/ascend/sglang:main-cann8.5.0-a3
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
# Stable release
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
# Daily build
docker pull quay.io/ascend/sglang:main-cann8.5.0-910b
```
</Tab>
</Tabs>
2. Build an image based on Dockerfile
```bash Command
# Clone the SGLang repository
@@ -165,7 +203,8 @@ docker build --build-arg TARGETARCH=<arch_tag> -t <image_name> -f npu.Dockerfile
#### Create Docker
__Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.
__Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
@@ -185,6 +224,32 @@ drun --env "HF_TOKEN=<secret>" \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci_manager --device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
# Add HF_TOKEN env for download model by SGLang.
drun --env "HF_TOKEN=<secret>" \
<image_name> \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```
</Tab>
</Tabs>
<Tip>
SGLang will serve on `http://127.0.0.1:30000` by default. You can change the host and port by `--host` and `--port` parameters.
</Tip>
## System Settings
### CPU performance power scheme
@@ -221,20 +286,51 @@ cat /proc/sys/vm/swappiness # shows 10
```bash Command
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--attention-backend ascend \
--host 127.0.0.1 \
--port 8000
```
#### PD Disaggregation Scene
1. Launch Prefill Server
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# PIP: recommended to config first Prefill Server IP
# PORT: one free port
# all sglang servers need to be config the same PIP and PORT,
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode prefill \
--disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8995 \
--attention-backend ascend \
--device npu \
--base-gpu-id 0 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8000
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
@@ -249,13 +345,39 @@ python3 -m sglang.launch_server \
--port 8000
```
</Tab>
</Tabs>
2. Launch Decode Server
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command
# PIP: recommended to config first Prefill Server IP
# PORT: one free port
# all sglang servers need to be config the same PIP and PORT,
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode decode \
--disaggregation-transfer-backend ascend \
--attention-backend ascend \
--device npu \
--base-gpu-id 1 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8001
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
@@ -269,6 +391,9 @@ python3 -m sglang.launch_server \
--port 8001
```
</Tab>
</Tabs>
3. Launch Router
```bash Command
python3 -m sglang_router.launch_router \
@@ -296,3 +421,78 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--sampling-backend ascend
```
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests.
### Which port to send requests to
The port you use depends on your deployment mode:
| Scenario | Where to send requests |
|---|---|
| Non-PD (single server) | The server's `--port` (e.g., `8000` in the examples above) |
| Non-PD (multi-node) | The primary node's (`--node-rank 0`) `--port`; do **not** send requests to worker nodes |
| PD disaggregation | The router's `--port` (e.g., `6688` in the examples above); do **not** send requests directly to prefill or decode servers |
<Tip>
SGLang serves on port `30000` by default if `--port` is not specified. The examples in this guide use explicit ports for clarity.
If you are using PD disaggregation, replace `8000` with your router's port (e.g., `6688`) in the following examples.
</Tip>
### Health Check
```bash Command
curl http://127.0.0.1:8000/health
```
A successful response returns HTTP 200 with an empty body.
### Generate (Native Endpoint)
```bash Command
curl http://127.0.0.1:8000/generate \
-H "Content-Type: application/json" \
-d '{
"text": "What is the capital of France?",
"sampling_params": {"temperature": 0, "max_new_tokens": 128}
}'
```
The expected output should contain "Paris".
### Chat Completions (OpenAI-Compatible)
```bash Command
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "What is the capital of France?"}]
}'
```
The expected output should contain "Paris".
### Multimodal Chat Completions
<Note>
The image URL in the example below references an external resource (`raw.githubusercontent.com`). Make sure the server has internet access so the image can be downloaded at inference time. Alternatively, you can use a locally accessible URL or base64-encoded image data.
</Note>
```bash Command
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3-VL-30B-A3B-Instruct",
"messages": [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
{"type": "text", "text": "Describe this image."}
]
}]
}'
```
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx
+31 -3
View File
@@ -16,11 +16,17 @@ This document describes how to perform accuracy evaluation for SGLang models run
## Environment Setup
<Warning>
Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
First, launch the SGLang environment using the provided container image:
<Tabs>
<Tab title="Atlas 800I A3">
```shell Command
# Atlas 800I A3 environment
export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-a3
export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
@@ -39,7 +45,29 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
$IMAGE
```
**For Atlas 800I A2 users**: Replace the image tag with `main-cann8.5.0-910b` and adjust device mappings from `/dev/davinci[8-15]` to `/dev/davinci[0-7]`.
</Tab>
<Tab title="Atlas 800I A2">
```shell Command
export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin \
--volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule \
--volume ~/.cache/:/root/.cache/ \
--entrypoint=bash \
$IMAGE
```
</Tab>
</Tabs>
---
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_deepseek_example.mdx
+9 -1
View File
@@ -6,6 +6,10 @@ metatags:
## Running DeepSeek-V3
<Warning>
If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3.
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
@@ -271,7 +275,11 @@ python -m sglang_router.launch_router \
--mini-lb
```
### test gsm8k
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
### Test GSM8K
```python Test GSM8K
from types import SimpleNamespace
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx
+60
View File
@@ -221,3 +221,63 @@ python -c "import sgl_kernel_npu; print(sgl_kernel_npu.__path__)"
rm -rf sgl-kernel-npu
```
## 6. `[Errno 101] Network is unreachable` when downloading HuggingFace datasets
### Error message
```text highlight=1-2
'[Errno 101] Network is unreachable' thrown while requesting HEAD https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
Retrying in 1s [Retry 1/5].
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 2353, in <module>
run_benchmark(args)
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 1848, in run_benchmark
input_requests = get_dataset(args, tokenizer, model_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/__init__.py", line 44, in get_dataset
return dataset.load(tokenizer=tokenizer, model_id=model_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 45, in load
return sample_random_requests(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 89, in sample_random_requests
dataset_path = download_and_cache_hf_file(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/utils.py", line 98, in download_and_cache_hf_file
return hf_hub_download(repo_id=repo_id, filename=filename, repo_type=repo_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
RuntimeError: Cannot send a request, as the client has been closed.
[ERROR] 2026-05-18-11:58:31 (PID:215, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```
### Cause
The machine cannot directly access the HuggingFace server due to network restrictions (firewall, proxy, or regional access limitations).
### Solution
- **Use an HF mirror site** — set the `HF_ENDPOINT` environment variable to a mirror (e.g., hf-mirror.com):
```bash
export HF_ENDPOINT=https://hf-mirror.com
```
- **Use a proxy** — if you have an HTTP proxy available:
```bash
export http_proxy=http://your-proxy:port
export https_proxy=http://your-proxy:port
```
- **Download the dataset manually** — use a machine with network access to download the file, then transfer it to the target machine. Use `--dataset-path` to specify the local file path:
```bash
python -m sglang.bench_serving \
--dataset-name random \
--dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
...
```
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5_examples.mdx
+48 -7
View File
@@ -11,6 +11,10 @@ The GLM (General Language Model) series is an open-source bilingual large langua
### Model Weight
<Warning>
If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
- `GLM-5.0`(BF16 version): [Download model weight](https://www.modelscope.cn/models/ZhipuAI/GLM-5).
- `GLM-5.0-w4a8`(Quantized version without mtp): [Download model weight](https://modelers.cn/models/Eco-Tech/GLM-5-w4a8).
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively.
@@ -18,15 +22,18 @@ The GLM (General Language Model) series is an open-source bilingual large langua
### Installation
<Warning>
Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space.
</Warning>
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command
#Atlas 800 A3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-glm5
#Atlas 800 A2
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-glm5
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
#start container
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
@@ -53,9 +60,39 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
```
</Tab>
</Tabs>
### Best Practices
Note: Using this image for **best practices**, you need to update transformers to version 5.3.0
```
@@ -195,7 +232,11 @@ done
### Prefill-Decode Disaggregation
Not test yet.
Not tested yet.
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
### Using Benchmark
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx
+42 -5
View File
@@ -13,14 +13,23 @@ metatags:
## Setup environment using container
__Notice:__ The following commands are based on Atlas 800I A3 machines. If you are using Atlas 800I A2, some changes are needed.
<Warning>
Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
- The image tag needs to be `main-cann8.5.0-a3` for Atlas 800I A3 and `main-cann8.5.0-910b` for Atlas 800I A2.
- The device mapping in `docker run` command needs to be changed to `davinci[0-7]` for Atlas 800I A2.
<Tip>
We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.10-npu.rc1-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann8.5.0-a3`) if you need the latest development changes.
</Tip>
<Tabs>
<Tab title="Atlas 800I A3">
```shell Command
# For Atlas 800I A3
export IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3
# Stable release
export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
# Daily build
export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-a3
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
@@ -39,6 +48,34 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
$IMAGE
```
</Tab>
<Tab title="Atlas 800I A2">
```shell Command
# Stable release
export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
# Daily build
export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-910b
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin \
--volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule \
--volume ~/.cache/:/root/.cache/ \
--entrypoint=bash \
$IMAGE
```
</Tab>
</Tabs>
## Usage
The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version.
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx
+48 -11
View File
@@ -7,15 +7,18 @@ metatags:
### Installation
<Warning>
Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
<Tabs>
<Tab title="Atlas 800I A3">
```bash Command
#Atlas 800 A3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3
#Atlas 800 A2
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-910b
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
#start container
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
@@ -42,9 +45,39 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${tag}
quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
```
</Tab>
<Tab title="Atlas 800I A2">
```bash Command
docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.10-npu.rc1-910b
```
</Tab>
</Tabs>
## Deployment
### Single-node Deployment
@@ -54,7 +87,7 @@ Run the following script to execute online inference.
#### Qwen3.5 397B
<Tip>
Recommended model: `Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp`
Recommended model: [`Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp)
</Tip>
```bash Command
@@ -101,7 +134,7 @@ python3 -m sglang.launch_server \
#### Qwen3.5 122B
<Tip>
Recommended model: `Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp`
Recommended model: [`Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp)
</Tip>
```bash Command
@@ -148,7 +181,7 @@ python3 -m sglang.launch_server \
#### Qwen3.5 35B
<Tip>
Recommended model: `Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp`
Recommended model: [`Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp)
</Tip>
```bash Command
@@ -195,7 +228,7 @@ python3 -m sglang.launch_server \
#### Qwen3.5 27B
<Tip>
Recommended model: `Eco-Tech/Qwen3.5-27B-w8a8-mtp`
Recommended model: [`Eco-Tech/Qwen3.5-27B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-27B-w8a8-mtp)
</Tip>
```bash Command
@@ -240,7 +273,11 @@ python3 -m sglang.launch_server \
### Prefill-Decode Disaggregation
Not test yet.
Not tested yet.
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, Multimodal Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
### Using Benchmark
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_examples.mdx
+8
View File
@@ -5,6 +5,10 @@ metatags:
---
## Qwen3 examples
<Warning>
If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
### Running Qwen3
#### Running Qwen3-32B on 1 x Atlas 800I A3.
@@ -204,3 +208,7 @@ python -m sglang.launch_server \
--model-path Qwen/Qwen3-VL-8B-Instruct \
--mem-fraction-static 0.8
```
## Testing the Service
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).