From 1f7bf155c3a27ba87ce896ee581d6c1d28854038 Mon Sep 17 00:00:00 2001 From: amote-i <49533125+amote-i@users.noreply.github.com> Date: Tue, 19 May 2026 16:22:22 +0800 Subject: [PATCH] [NPU] [DOCS] Improved the usability of Ascend NPU documents (#25735) --- .../ascend-npus/ascend_npu.mdx | 244 ++++++++++++++++-- .../ascend_npu_accuracy_evaluation.mdx | 34 ++- .../ascend_npu_deepseek_example.mdx | 10 +- .../ascend-npus/ascend_npu_faq.mdx | 60 +++++ .../ascend-npus/ascend_npu_glm5_examples.mdx | 55 +++- .../ascend-npus/ascend_npu_quick_start.mdx | 47 +++- .../ascend_npu_qwen3_5_examples.mdx | 59 ++++- .../ascend-npus/ascend_npu_qwen3_examples.mdx | 8 + 8 files changed, 468 insertions(+), 49 deletions(-) diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx index c4bf6eb58..c78c3998b 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx @@ -53,19 +53,29 @@ You can install SGLang using any of the methods below. Please go through `System ### Obtain CANN Image + +Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space. + + You can obtain the dependency of a specified version of CANN through an image. - -CANN images and SGLang images are hosted at different registry addresses. Make sure to pull them from the correct location. - + + ```bash Command -# for Atlas 800I A3 and Ubuntu OS docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11 -# for Atlas 800I A2 and Ubuntu OS +``` + + + + +```bash Command docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11 ``` + + + ## Preparing the Running Environment ### Method 1: Installing from source with prerequisites @@ -142,14 +152,42 @@ pip install -e python[all_npu] ### Method 2: Using Docker Image #### Obtain Image You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image. + + +Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. + + 1. Download SGLang image + + +We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.10-npu.rc1-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann8.5.0-a3`) if you need the latest development changes. + + + + + ```bash Command -dockerhub: docker.io/lmsysorg/sglang:$tag -# Main-based tag, change main to specific version like v0.5.6, -# you can get image for specific version -Atlas 800I A3 : {main}-cann8.5.0-a3 -Atlas 800I A2: {main}-cann8.5.0-910b +# Stable release +docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3 + +# Daily build +docker pull quay.io/ascend/sglang:main-cann8.5.0-a3 ``` + + + + +```bash Command +# Stable release +docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b + +# Daily build +docker pull quay.io/ascend/sglang:main-cann8.5.0-910b +``` + + + + 2. Build an image based on Dockerfile ```bash Command # Clone the SGLang repository @@ -165,7 +203,8 @@ docker build --build-arg TARGETARCH= -t -f npu.Dockerfile #### Create Docker __Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters. -__Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container. + + ```bash Command alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ @@ -185,6 +224,32 @@ drun --env "HF_TOKEN=" \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend ``` + + + +```bash Command +alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ + --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ + --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ + --device=/dev/davinci_manager --device=/dev/hisi_hdc \ + --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ + --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ + --volume /etc/ascend_install.info:/etc/ascend_install.info \ + --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/' + +# Add HF_TOKEN env for download model by SGLang. +drun --env "HF_TOKEN=" \ + \ + python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend +``` + + + + + +SGLang will serve on `http://127.0.0.1:30000` by default. You can change the host and port by `--host` and `--port` parameters. + + ## System Settings ### CPU performance power scheme @@ -221,20 +286,51 @@ cat /proc/sys/vm/swappiness # shows 10 ```bash Command # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 -python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend +python3 -m sglang.launch_server \ + --model-path meta-llama/Llama-3.1-8B-Instruct \ + --attention-backend ascend \ + --host 127.0.0.1 \ + --port 8000 ``` #### PD Disaggregation Scene 1. Launch Prefill Server + + + + ```bash Command # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 -# PIP: recommended to config first Prefill Server IP -# PORT: one free port -# all sglang servers need to be config the same PIP and PORT, -export ASCEND_MF_STORE_URL="tcp://PIP:PORT" -# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter +# PREFILL_IP: IP address of the first Prefill Server +# FREE_PORT: any available port +# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT +export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" +python3 -m sglang.launch_server \ + --model-path meta-llama/Llama-3.1-8B-Instruct \ + --disaggregation-mode prefill \ + --disaggregation-transfer-backend ascend \ + --disaggregation-bootstrap-port 8995 \ + --attention-backend ascend \ + --device npu \ + --base-gpu-id 0 \ + --tp-size 1 \ + --host 127.0.0.1 \ + --port 8000 +``` + + + + +```bash Command +# Enabling CPU Affinity +export SGLANG_SET_CPU_AFFINITY=1 + +# PREFILL_IP: IP address of the first Prefill Server +# FREE_PORT: any available port +# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT +export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ @@ -249,13 +345,39 @@ python3 -m sglang.launch_server \ --port 8000 ``` + + + 2. Launch Decode Server + + + + ```bash Command -# PIP: recommended to config first Prefill Server IP -# PORT: one free port -# all sglang servers need to be config the same PIP and PORT, -export ASCEND_MF_STORE_URL="tcp://PIP:PORT" -# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter +# PREFILL_IP: IP address of the first Prefill Server +# FREE_PORT: any available port +# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT +export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" +python3 -m sglang.launch_server \ + --model-path meta-llama/Llama-3.1-8B-Instruct \ + --disaggregation-mode decode \ + --disaggregation-transfer-backend ascend \ + --attention-backend ascend \ + --device npu \ + --base-gpu-id 1 \ + --tp-size 1 \ + --host 127.0.0.1 \ + --port 8001 +``` + + + + +```bash Command +# PREFILL_IP: IP address of the first Prefill Server +# FREE_PORT: any available port +# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT +export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ @@ -269,6 +391,9 @@ python3 -m sglang.launch_server \ --port 8001 ``` + + + 3. Launch Router ```bash Command python3 -m sglang_router.launch_router \ @@ -296,3 +421,78 @@ python3 -m sglang.launch_server \ --enable-multimodal \ --sampling-backend ascend ``` + +## Testing the Service + +Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. + +### Which port to send requests to + +The port you use depends on your deployment mode: + +| Scenario | Where to send requests | +|---|---| +| Non-PD (single server) | The server's `--port` (e.g., `8000` in the examples above) | +| Non-PD (multi-node) | The primary node's (`--node-rank 0`) `--port`; do **not** send requests to worker nodes | +| PD disaggregation | The router's `--port` (e.g., `6688` in the examples above); do **not** send requests directly to prefill or decode servers | + + +SGLang serves on port `30000` by default if `--port` is not specified. The examples in this guide use explicit ports for clarity. + +If you are using PD disaggregation, replace `8000` with your router's port (e.g., `6688`) in the following examples. + + +### Health Check + +```bash Command +curl http://127.0.0.1:8000/health +``` + +A successful response returns HTTP 200 with an empty body. + +### Generate (Native Endpoint) + +```bash Command +curl http://127.0.0.1:8000/generate \ + -H "Content-Type: application/json" \ + -d '{ + "text": "What is the capital of France?", + "sampling_params": {"temperature": 0, "max_new_tokens": 128} + }' +``` + +The expected output should contain "Paris". + +### Chat Completions (OpenAI-Compatible) + +```bash Command +curl http://127.0.0.1:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "meta-llama/Llama-3.1-8B-Instruct", + "messages": [{"role": "user", "content": "What is the capital of France?"}] + }' +``` + +The expected output should contain "Paris". + +### Multimodal Chat Completions + + +The image URL in the example below references an external resource (`raw.githubusercontent.com`). Make sure the server has internet access so the image can be downloaded at inference time. Alternatively, you can use a locally accessible URL or base64-encoded image data. + + +```bash Command +curl http://127.0.0.1:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3-VL-30B-A3B-Instruct", + "messages": [{ + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, + {"type": "text", "text": "Describe this image."} + ] + }] + }' +``` diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx index 38ae57679..7c5292bfd 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx @@ -16,11 +16,17 @@ This document describes how to perform accuracy evaluation for SGLang models run ## Environment Setup + +Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. + + First, launch the SGLang environment using the provided container image: + + + ```shell Command -# Atlas 800I A3 environment -export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-a3 +export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-a3 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ @@ -39,7 +45,29 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ $IMAGE ``` -**For Atlas 800I A2 users**: Replace the image tag with `main-cann8.5.0-910b` and adjust device mappings from `/dev/davinci[8-15]` to `/dev/davinci[0-7]`. + + + +```shell Command +export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-910b + +docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ + --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ + --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ + --device=/dev/davinci_manager \ + --device=/dev/hisi_hdc \ + --volume /usr/local/sbin:/usr/local/sbin \ + --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ + --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ + --volume /etc/ascend_install.info:/etc/ascend_install.info \ + --volume /var/queue_schedule:/var/queue_schedule \ + --volume ~/.cache/:/root/.cache/ \ + --entrypoint=bash \ + $IMAGE +``` + + + --- diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_deepseek_example.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_deepseek_example.mdx index dc99df7dd..78e1cb725 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_deepseek_example.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_deepseek_example.mdx @@ -6,6 +6,10 @@ metatags: ## Running DeepSeek-V3 + +If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. + + ### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3. W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8). @@ -271,7 +275,11 @@ python -m sglang_router.launch_router \ --mini-lb ``` -### test gsm8k +## Testing the Service + +Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). + +### Test GSM8K ```python Test GSM8K from types import SimpleNamespace diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx index 7da46609f..c33e233b3 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx @@ -221,3 +221,63 @@ python -c "import sgl_kernel_npu; print(sgl_kernel_npu.__path__)" rm -rf sgl-kernel-npu ``` + +## 6. `[Errno 101] Network is unreachable` when downloading HuggingFace datasets + +### Error message + +```text highlight=1-2 +'[Errno 101] Network is unreachable' thrown while requesting HEAD https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json +Retrying in 1s [Retry 1/5]. +Traceback (most recent call last): + File "", line 198, in _run_module_as_main + File "", line 88, in _run_code + File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 2353, in + run_benchmark(args) + File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 1848, in run_benchmark + input_requests = get_dataset(args, tokenizer, model_id) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/__init__.py", line 44, in get_dataset + return dataset.load(tokenizer=tokenizer, model_id=model_id) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 45, in load + return sample_random_requests( + ^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 89, in sample_random_requests + dataset_path = download_and_cache_hf_file( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/utils.py", line 98, in download_and_cache_hf_file + return hf_hub_download(repo_id=repo_id, filename=filename, repo_type=repo_type) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ... +RuntimeError: Cannot send a request, as the client has been closed. +[ERROR] 2026-05-18-11:58:31 (PID:215, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception +``` + +### Cause + +The machine cannot directly access the HuggingFace server due to network restrictions (firewall, proxy, or regional access limitations). + +### Solution + +- **Use an HF mirror site** — set the `HF_ENDPOINT` environment variable to a mirror (e.g., hf-mirror.com): + + ```bash + export HF_ENDPOINT=https://hf-mirror.com + ``` + +- **Use a proxy** — if you have an HTTP proxy available: + + ```bash + export http_proxy=http://your-proxy:port + export https_proxy=http://your-proxy:port + ``` + +- **Download the dataset manually** — use a machine with network access to download the file, then transfer it to the target machine. Use `--dataset-path` to specify the local file path: + + ```bash + python -m sglang.bench_serving \ + --dataset-name random \ + --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \ + ... + ``` diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5_examples.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5_examples.mdx index 333f07c8c..13aa28924 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5_examples.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5_examples.mdx @@ -11,6 +11,10 @@ The GLM (General Language Model) series is an open-source bilingual large langua ### Model Weight + +If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. + + - `GLM-5.0`(BF16 version): [Download model weight](https://www.modelscope.cn/models/ZhipuAI/GLM-5). - `GLM-5.0-w4a8`(Quantized version without mtp): [Download model weight](https://modelers.cn/models/Eco-Tech/GLM-5-w4a8). - You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively. @@ -18,15 +22,18 @@ The GLM (General Language Model) series is an open-source bilingual large langua ### Installation + +Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. + + The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. + + + ```bash Command -#Atlas 800 A3 -docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-glm5 -#Atlas 800 A2 -docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-glm5 +docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3 -#start container docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ @@ -53,9 +60,39 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG} +quay.io/ascend/sglang:v0.5.10-npu.rc1-a3 ``` + + + +```bash Command +docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b + +docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ +--privileged=true --net=host \ +-v /var/queue_schedule:/var/queue_schedule \ +-v /etc/ascend_install.info:/etc/ascend_install.info \ +-v /usr/local/sbin:/usr/local/sbin \ +-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ +-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ +--device=/dev/davinci0:/dev/davinci0 \ +--device=/dev/davinci1:/dev/davinci1 \ +--device=/dev/davinci2:/dev/davinci2 \ +--device=/dev/davinci3:/dev/davinci3 \ +--device=/dev/davinci4:/dev/davinci4 \ +--device=/dev/davinci5:/dev/davinci5 \ +--device=/dev/davinci6:/dev/davinci6 \ +--device=/dev/davinci7:/dev/davinci7 \ +--device=/dev/davinci_manager:/dev/davinci_manager \ +--device=/dev/hisi_hdc:/dev/hisi_hdc \ +--entrypoint=bash \ +quay.io/ascend/sglang:v0.5.10-npu.rc1-910b +``` + + + + ### Best Practices Note: Using this image for **best practices**, you need to update transformers to version 5.3.0 ``` @@ -195,7 +232,11 @@ done ### Prefill-Decode Disaggregation -Not test yet. +Not tested yet. + +## Testing the Service + +Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). ### Using Benchmark diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx index 3b4545339..7bd688634 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx @@ -13,14 +13,23 @@ metatags: ## Setup environment using container -__Notice:__ The following commands are based on Atlas 800I A3 machines. If you are using Atlas 800I A2, some changes are needed. + +Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. + -- The image tag needs to be `main-cann8.5.0-a3` for Atlas 800I A3 and `main-cann8.5.0-910b` for Atlas 800I A2. -- The device mapping in `docker run` command needs to be changed to `davinci[0-7]` for Atlas 800I A2. + +We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.10-npu.rc1-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann8.5.0-a3`) if you need the latest development changes. + + + + ```shell Command -# For Atlas 800I A3 -export IMAGE=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3 +# Stable release +export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-a3 + +# Daily build +export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-a3 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ @@ -39,6 +48,34 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ $IMAGE ``` + + + +```shell Command +# Stable release +export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-910b + +# Daily build +export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-910b + +docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ + --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ + --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ + --device=/dev/davinci_manager \ + --device=/dev/hisi_hdc \ + --volume /usr/local/sbin:/usr/local/sbin \ + --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ + --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ + --volume /etc/ascend_install.info:/etc/ascend_install.info \ + --volume /var/queue_schedule:/var/queue_schedule \ + --volume ~/.cache/:/root/.cache/ \ + --entrypoint=bash \ + $IMAGE +``` + + + + ## Usage The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx index 93144929d..eb054f778 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx @@ -7,15 +7,18 @@ metatags: ### Installation + +Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. + + The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. + + + ```bash Command -#Atlas 800 A3 -docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3 -#Atlas 800 A2 -docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-910b +docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-a3 -#start container docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ @@ -42,9 +45,39 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ -swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${tag} +quay.io/ascend/sglang:v0.5.10-npu.rc1-a3 ``` + + + +```bash Command +docker pull quay.io/ascend/sglang:v0.5.10-npu.rc1-910b + +docker run -itd --shm-size=16g --privileged=true --name ${NAME} \ +--privileged=true --net=host \ +-v /var/queue_schedule:/var/queue_schedule \ +-v /etc/ascend_install.info:/etc/ascend_install.info \ +-v /usr/local/sbin:/usr/local/sbin \ +-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ +-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ +--device=/dev/davinci0:/dev/davinci0 \ +--device=/dev/davinci1:/dev/davinci1 \ +--device=/dev/davinci2:/dev/davinci2 \ +--device=/dev/davinci3:/dev/davinci3 \ +--device=/dev/davinci4:/dev/davinci4 \ +--device=/dev/davinci5:/dev/davinci5 \ +--device=/dev/davinci6:/dev/davinci6 \ +--device=/dev/davinci7:/dev/davinci7 \ +--device=/dev/davinci_manager:/dev/davinci_manager \ +--device=/dev/hisi_hdc:/dev/hisi_hdc \ +--entrypoint=bash \ +quay.io/ascend/sglang:v0.5.10-npu.rc1-910b +``` + + + + ## Deployment ### Single-node Deployment @@ -54,7 +87,7 @@ Run the following script to execute online inference. #### Qwen3.5 397B -Recommended model: `Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp` +Recommended model: [`Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp) ```bash Command @@ -101,7 +134,7 @@ python3 -m sglang.launch_server \ #### Qwen3.5 122B -Recommended model: `Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp` +Recommended model: [`Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp) ```bash Command @@ -148,7 +181,7 @@ python3 -m sglang.launch_server \ #### Qwen3.5 35B -Recommended model: `Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp` +Recommended model: [`Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp) ```bash Command @@ -195,7 +228,7 @@ python3 -m sglang.launch_server \ #### Qwen3.5 27B -Recommended model: `Eco-Tech/Qwen3.5-27B-w8a8-mtp` +Recommended model: [`Eco-Tech/Qwen3.5-27B-w8a8-mtp`](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-27B-w8a8-mtp) ```bash Command @@ -240,7 +273,11 @@ python3 -m sglang.launch_server \ ### Prefill-Decode Disaggregation -Not test yet. +Not tested yet. + +## Testing the Service + +Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, Multimodal Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). ### Using Benchmark diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_examples.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_examples.mdx index 0bb0947ad..9073096d6 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_examples.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_examples.mdx @@ -5,6 +5,10 @@ metatags: --- ## Qwen3 examples + +If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. + + ### Running Qwen3 #### Running Qwen3-32B on 1 x Atlas 800I A3. @@ -204,3 +208,7 @@ python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --mem-fraction-static 0.8 ``` + +## Testing the Service + +Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).