minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[NPU] [DOC] Fix issues about npu docs found by aidd (#31302)

Browse Source
This commit is contained in:
amote-i
2026-07-20 16:31:27 +08:00
committed by GitHub
parent fce5c75a30
commit 97e0647bdc
29 changed files with 429 additions and 388 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
+3 -3
View File
@@ -107,7 +107,7 @@ conda config --set show_channel_urls yes
conda config --remove channels defaults
```
Edit the system-level conda config to remove any hardcoded defaults, e.g. vi ~/miniconda3/.condarc
Edit the system-level conda config to remove any hardcoded defaults, e.g., vi ~/miniconda3/.condarc
Then remove the failed environment and recreate it:
```bash Command
@@ -140,7 +140,7 @@ installed in addition to MemFabric-Hybrid.
pip install memfabric-zbal==1.1.1
```
#### Pytorch and Pytorch Framework Adaptor on Ascend
#### PyTorch and PyTorch Framework Adaptor on Ascend
```bash Command
PYTORCH_VERSION=2.10.0
@@ -242,7 +242,7 @@ git clone https://github.com/sgl-project/sglang.git
cd sglang/docker
# Build the docker image
# Replace <arch_tag> with the target architecture, e.g. amd64, arm64.
# Replace <arch_tag> with the target architecture, e.g., amd64, arm64.
# Optional build arguments:
# --build-arg DEVICE_TYPE=910b # Required for Atlas 800I A2
# --build-arg APTMIRROR=<mirror_url> # Use a custom APT mirror to improve download speed
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx
+1 -1
View File
@@ -23,7 +23,7 @@ on Ascend hardware.
## Directory Structure
<Note>
The identifiers `sgl_kenel_npu_ops.h`, `KernalHelloworld`, and `retrive_*` (e.g. `retrive_index`,
The identifiers `sgl_kenel_npu_ops.h`, `KernalHelloworld`, and `retrive_*` (e.g., `retrive_index`,
`retrive_next_token`, `retrive_next_sibling`) in this guide match the spelling used in the
upstream [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository and are kept verbatim for consistency.
</Note>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx
+5 -1
View File
@@ -745,7 +745,11 @@ Separates the prefill (P) and decode (D) phases onto different device groups, al
### Quantization (`--quantization`)
Reduces model weight and activation precision (e.g., W8A8, W4A8, W4A16) to decrease memory usage and increase throughput. On Ascend, the natively supported and highly optimized quantization method is ModelSlim (supporting W4A4, W8A8, W4A8 dynamic/static). Support for other community formats (such as AWQ, GPTQ, Auto-round, and Compressed-tensors) depends on the availability of specific Ascend custom kernels in your environment; see quantization for details and compatibility matrices.
Reduces model weight and activation precision (e.g., W8A8, W4A8, W4A16) to decrease memory usage and increase throughput.
On Ascend, the natively supported and highly optimized quantization method is ModelSlim (supporting W4A4, W8A8, W4A8 dynamic/static).
Support for other community formats (such as AWQ, GPTQ, Auto-round, and Compressed-tensors) depends on the availability
of specific Ascend custom kernels in your environment;
see [Quantization on Ascend](/docs/hardware-platforms/ascend-npus/ascend_npu_quantization) for details and compatibility matrices.
### Chunked Prefill (`--chunked-prefill-size`)
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx
+2 -2
View File
@@ -221,7 +221,7 @@ sending `/stop_profile`.
<Note>
`bench_serving --profile` creates a timestamp subdirectory inside
`--profile-output-dir` (e.g. `<output_dir>/<timestamp>/`). The output path is
`--profile-output-dir` (e.g., `<output_dir>/<timestamp>/`). The output path is
shown in the server log as `Profiling done. Traces are saved to: <path>`.
</Note>
@@ -393,7 +393,7 @@ set of supported parameters:
<tr>
<td><code>profile_stages</code></td>
<td>
Stages to profile, e.g. <code>["prefill", "decode"]</code>.
Stages to profile, e.g., <code>["prefill", "decode"]</code>.
Requires <code>profile_by_stage</code>
</td>
<td>None</td>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx
+4 -4
View File
@@ -307,7 +307,7 @@ Compressed-tensors (LLM Compressor) on Ascend support:
**Usage Examples:**
- Dense model (e.g. Qwen3-14B-Q4_K_M.gguf):
- Dense model (e.g., Qwen3-14B-Q4_K_M.gguf):
```bash Command
python3 -m sglang.launch_server \
@@ -317,7 +317,7 @@ python3 -m sglang.launch_server \
--mem-fraction-static 0.7 --tp-size 2
```
- MoE model (e.g. Qwen3-30B-A3B-Q4_K_M.gguf):
- MoE model (e.g., Qwen3-30B-A3B-Q4_K_M.gguf):
```bash Command
python3 -m sglang.launch_server \
@@ -333,7 +333,7 @@ python3 -m sglang.launch_server \
> - MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation.
> - TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models.
**MXFP8 for LLM dense models (e.g. Qwen3 / Qwen3.5):**
**MXFP8 for LLM dense models (e.g., Qwen3 / Qwen3.5):**
LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend A5 series or newer (`npu_dynamic_mx_quant` is not available on A2 / A3).
@@ -363,7 +363,7 @@ python3 -m sglang.launch_server \
> - Offline path: `ModelSlimMXFP8Scheme` loads `float8_e4m3fn` weights + `float8_e8m0fnu` block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (`.data` assignment) — calling `.contiguous()` would physically reorder the pre-quantized layout and break the block-scale mapping.
> - MoE MXFP8 (FusedMoE/TP) for LLMs is tracked separately and not part of this PR.
**MXFP4 W4A8 for LLM dense models (e.g. Qwen3 / Qwen3.5):**
**MXFP4 W4A8 for LLM dense models (e.g., Qwen3 / Qwen3.5):**
LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend A5 series or newer.
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx
+35 -17
View File
@@ -17,25 +17,43 @@ see [SGLang installation with NPUs support](/docs/hardware-platforms/ascend-npus
- Atlas 800I A2 inference series (Atlas 800I A2)
- Atlas 800I A3 inference series (Atlas 800I A3)
To identify your device, run `npu-smi info -l`: A3 reports `Chip Count: 2` per NPU, while A2 reports `Chip Count: 1` per NPU.
For hardware details, see the [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
### Docker
Ensure Docker is installed and the Docker daemon is running on your host machine. Verify with:
```bash
docker --version && docker info
```
If Docker is not installed, follow the [official Docker installation guide](https://docs.docker.com/engine/install/) for
your operating system.
## Setup environment using container
<Warning>
Ensure sufficient disk space before proceeding. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
Ensure sufficient disk space before proceeding. Run `df -h` to check the available disk space. The Docker image requires
at least **30GB** of free space. If you need to download model weights, check the model size at
[ModelScope](https://www.modelscope.cn/models) to reserve enough space.
</Warning>
<Tip>
We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.13.post1-cann9.0.0-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes.
We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.13.post1-cann9.0.0-a3`)
if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes.
</Tip>
If you have already downloaded model weights to a local path (e.g., `/path/to/model`), mount the path into the container by
adding `--volume /path/to/model:/path/to/model` to the `docker run` command below.
<Tabs>
<Tab title="Atlas 800I A3">
```shell Command
# Stable release
export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3
# Daily build
export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3
# Choose one (uncomment the line you want):
export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3 # Stable release
# export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3 # Daily build
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
@@ -58,11 +76,9 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
<Tab title="Atlas 800I A2">
```shell Command
# Stable release
export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b
# Daily build
export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b
# Choose one (uncomment the line you want):
export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b # Stable release
# export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b # Daily build
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
@@ -88,7 +104,8 @@ The SGLang server is installed in the container by default. You can use `pip sho
### Start SGLang server
SGLang will automatically download the model from Hugging Face.
SGLang will automatically download the model from Hugging Face. If the model is already downloaded to a local path
(and has been mounted into the container), use that path directly like `--model-path /path/to/model`.
```shell Command
# Set HF_ENDPOINT to a mirror site if network is not available
@@ -102,7 +119,7 @@ export HF_TOKEN=<secret>
sglang serve --model-path Qwen/Qwen2.5-7B-Instruct --attention-backend ascend &
```
If you see output like the following, the server is running.
Server startup may take several minutes. Once you see output like the following, the server is running.
```log Output
INFO: Waiting for application startup.
@@ -138,13 +155,14 @@ SGLANG_PID=$(pgrep -f "sglang serve")
kill -SIGINT $SGLANG_PID
```
The output should be like the following:
Wait a moment for the server to shut down gracefully. The output should be like the following:
```log Output
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [25310]
INFO: Finished server process [<SGLANG_PID>]
```
The server has now stopped. You can verify it with `ps -ef | grep sglang`, then exit the container by pressing `Ctrl+D`.
The server has now stopped. You can verify it with `ps -ef | grep sglang` — the expected output is nothing (no matching process),
then exit the container by pressing `Ctrl+D`.
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_reference.mdx
+1 -1
View File
@@ -87,7 +87,7 @@ PD disaggregation separates inference into two stages running on different nodes
| **1P2D** | 1 prefill node + 2 decode nodes |
You will see these in [Best Practice](/docs/hardware-platforms/ascend-npus/best_practice)
section headings, e.g. `W8A8 2P1D 32P` means "W8A8 quantization, 2 prefill nodes +
section headings, e.g., `W8A8 2P1D 32P` means "W8A8 quantization, 2 prefill nodes +
1 decode node, 32 cards total."
### Parallelism strategies
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx
+22 -22
View File
@@ -3,7 +3,7 @@ title: "Support Features on Ascend NPU"
metatags:
description: "Documentation for Support Features on Ascend NPU"
---
This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any
This section describes the basic functions and features supported by the Ascend NPU. If you encounter issues or have any
questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
If you want to know the meaning and usage of each parameter,
@@ -279,7 +279,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--quantization-param-path`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: str</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special For GPU</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special for GPU</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--kv-cache-dtype`</td>
@@ -297,37 +297,37 @@ click [Server Arguments](../../advanced_features/server_arguments).
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--modelopt-quant`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: str</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special For GPU</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special for GPU</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--modelopt-checkpoint-restore-path`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: str</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special For GPU</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special for GPU</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--modelopt-checkpoint-save-path`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: str</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special For GPU</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special for GPU</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--modelopt-export-path`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: str</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special For GPU</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special for GPU</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--quantize-and-serve`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag <br/> (set to enable)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special For GPU</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special for GPU</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--rl-quant-profile`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: str</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special For GPU</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special for GPU</td>
</tr>
</tbody>
</table>
@@ -1157,7 +1157,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
</tbody>
</table>
## LoRA(Restrict the use Qwen Series Models)
## LoRA (restricted to Qwen series models)
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -1180,13 +1180,13 @@ click [Server Arguments](../../advanced_features/server_arguments).
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-lora`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Bool flag <br/>(set to enable)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag <br/>(set to enable)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-lora-overlap-loading`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Bool flag <br/>(set to enable)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag <br/>(set to enable)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
@@ -1227,8 +1227,8 @@ click [Server Arguments](../../advanced_features/server_arguments).
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--lora-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`ascend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`ascend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`csgmv`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`,<br/> `csgmv`,<br/> `ascend`,<br/> `torch_native`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
@@ -1264,7 +1264,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
</tbody>
</table>
## Kernel Backends (Attention, Sampling, Grammar, GEMM)
## Kernel backends (attention, sampling, grammar, GEMM)
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -1741,7 +1741,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
</tbody>
</table>
## Mamba Cache
## Mamba cache
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -2544,7 +2544,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disaggregation-transfer-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`mooncake`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`ascend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`ascend` (default `mooncake` not supported on NPU, must be specified manually)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
@@ -2608,7 +2608,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-adaptive-dispatch-to-encoder`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag<br/> (set to enable adaptively dispatch)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag<br/> (set to enable adaptive dispatch)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
@@ -2733,7 +2733,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
</tbody>
</table>
## For PD-Multiplexing
## For PD-multiplexing
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -2774,7 +2774,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
</tbody>
</table>
## For Multi-Modal
## For multi-modal
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -2951,10 +2951,10 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
</tbody>
</table>
## Other Params
## Other params
The following parameters are not supported because the third-party components that depend on are not compatible with the
NPU, like Ktransformer, checkpoint-engine etc.
The following parameters are not supported because the required third-party components
(e.g., Ktransformer, checkpoint-engine) are not compatible with the NPU.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx
+1 -1
View File
@@ -608,7 +608,7 @@ You are welcome to enable various models based on your business requirements.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Alibaba-NLP/gme-Qwen2-VL-2B-Instruct</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>iic/gme-Qwen2-VL-2B-Instruct</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GME (Multimodal)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx
+6 -2
View File
@@ -64,7 +64,10 @@ Please note all your testing and benchmarking results in PR description.
### Benchmark
- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow [https://docs.sglang.io/developer_guide/benchmark_and_profiling.html](https://docs.sglang.io/developer_guide/benchmark_and_profiling.html) to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer).
- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md)
to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than
that from HF Transformer run. Similarly, follow the [benchmark and profiling guide](/docs/developer_guide/benchmark_and_profiling)
to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer).
- **(Optional) Other evals**: If you ran other evals, please note the results in PR description.
<Tip>
@@ -99,7 +102,8 @@ To port a model from vLLM to SGLang:
patterns, such as replacing CUDA kernels with `torch_npu` equivalents. The NPU backend is at
`sglang/srt/hardware_backend/npu/`.
Note: make sure you add your new model to the supported models list in the supported models documentation.
Note: make sure you add your new model to the supported models list in the
[supported models documentation](https://github.com/sgl-project/sglang/blob/main/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx).
## Registering an External Model Implementation
docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx
+4 -4
View File
@@ -538,7 +538,7 @@ python -m sglang_router.launch_router \
We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 115200` = `int(128000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 12799` = `int(128000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-question-len 12800` = `int(128000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.
```bash Command
@@ -550,7 +550,7 @@ python -m sglang.bench_serving \
--gsp-num-groups 1 \
--gsp-prompts-per-group 4 \
--gsp-system-prompt-len 115200 \
--gsp-question-len 12799 \
--gsp-question-len 12800 \
--gsp-output-len 1000 \
--max-concurrency 1 \
--num-prompts 4 \
@@ -931,7 +931,7 @@ python -m sglang_router.launch_router \
We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6399` = `int(64000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-question-len 6400` = `int(64000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.
```bash Command
@@ -943,7 +943,7 @@ python -m sglang.bench_serving \
--gsp-num-groups 1 \
--gsp-prompts-per-group 8 \
--gsp-system-prompt-len 57600 \
--gsp-question-len 6399 \
--gsp-question-len 6400 \
--gsp-output-len 1500 \
--max-concurrency 2 \
--num-prompts 8 \
docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx
+5 -5
View File
@@ -16,7 +16,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend
| --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms) |
| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1080p-30-out256-50ms) |
| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-2p-in64k-out1k-prefix90-50ms) |
| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in64k-out1k-prefix90-50ms) |
| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) |
| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) |
| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 128k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) |
@@ -224,13 +224,13 @@ python -m sglang.bench_serving \
--random-range-ratio 1
```
### Qwen3.6-27B 2P IN64K OUT1K PREFIX90 50ms
### Qwen3.6-27B 1P IN64K OUT1K PREFIX90 50ms
**Model**: Qwen3.6-27B
**Hardware**: Atlas 800I A3
**Cards**: 2
**Cards**: 1
**Deploy Mode**: PD Mixed
@@ -307,7 +307,7 @@ python3 -m sglang.launch_server \
We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6399` = `int(64000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-question-len 6400` = `int(64000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.
```bash Command
@@ -319,7 +319,7 @@ python -m sglang.bench_serving \
--gsp-num-groups 1 \
--gsp-prompts-per-group 80 \
--gsp-system-prompt-len 57600 \
--gsp-question-len 6399 \
--gsp-question-len 6400 \
--gsp-output-len 1000 \
--max-concurrency 20 \
--num-prompts 80 \
docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx
-103
View File
@@ -23,7 +23,6 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms) |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms) |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) |
@@ -327,108 +326,6 @@ python -m sglang.bench_serving \
--random-range-ratio 1
```
### Qwen3.6-35B-A3B 1P IN128K OUT1K PREFIX90 50ms
**Model**: Qwen3.6-35B-A3B
**Hardware**: Atlas 800I A3
**Cards**: 1
**Deploy Mode**: PD Mixed
**Quantization**: BF16
**Dataset**: 128k+1k (90% prefix cache hit rate)
**TPOT**: 50ms
#### Model Deployment
```bash Command
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# HCCL_SOCKET_IFNAME: network interface name for HCCL
# GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================
MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 2 \
--nnodes 1 \
--attention-backend ascend \
--device npu \
--chunked-prefill-size 16384 \
--max-prefill-tokens 65536 \
--trust-remote-code \
--enable-prefill-delayer \
--mamba-radix-cache-strategy extra_buffer \
--max-running-requests 103 \
--max-mamba-cache-size 85 \
--mem-fraction-static 0.85 \
--cuda-graph-bs 2 4 8 16 32 48 64 80 96 103 \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
```
#### Benchmark
We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6399` = `int(64000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.
```bash Command
python -m sglang.bench_serving \
--dataset-name generated-shared-prefix \
--backend sglang \
--host 127.0.0.1 \
--port 6688 \
--gsp-num-groups 1 \
--gsp-prompts-per-group 412 \
--gsp-system-prompt-len 57600 \
--gsp-question-len 6399 \
--gsp-output-len 1000 \
--max-concurrency 103 \
--num-prompts 412 \
--request-rate inf
```
### Qwen3.6-35B-A3B 1P IN254K OUT1K
**Model**: Qwen3.6-35B-A3B
docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx
+3 -3
View File
@@ -14,7 +14,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-20ms) |
| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-1p-in3k5-out1k5-20ms) |
| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 6k+1.5k | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) |
### High Throughput
@@ -27,13 +27,13 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend
<a id="single-node-pd-mixed" title="Referenced by external docs. Verify before removing."></a>
### Qwen3-Next-80B-A3B-Instruct W8A8 2P IN3K5 OUT1K5 20ms
### Qwen3-Next-80B-A3B-Instruct W8A8 1P IN3K5 OUT1K5 20ms
**Model**: Qwen3-Next-80B-A3B-Instruct
**Hardware**: Atlas 800I A3
**Cards**: 2
**Cards**: 1
**Deploy Mode**: PD Mixed
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx
+17 -9
View File
@@ -1,7 +1,7 @@
---
title: "DeepSeek-R1"
metatags:
description: "Deploy DeepSeek-R1 model with SGLang on Ascend NPUs, including single-node and multi-node PD disaggregation modes."
description: "Deploy DeepSeek-R1 model with SGLang on Ascend NPUs, including single-node PD mixed and multi-node PD disaggregation modes."
---
## Introduction
@@ -27,7 +27,7 @@ v0.5.13 or a later version.
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 4 8 20 21 22` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 20 21 22` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -61,17 +61,25 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
Before downloading model weights, check the model size to reserve enough disk space.
</Warning>
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [DeepSeek-R1-0528-W4A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8) (Quantized version)
- [DeepSeek-R1-0528-W8A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w8a8) (Quantized version)
- [DeepSeek-R1-0528-W4A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8) (Quantized version, 376GB)
- [DeepSeek-R1-0528-W8A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w8a8) (Quantized version, 647GB)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
<Info>
The W4A8 variant (376GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2
node or 8 dies on A3 (4 cards).
It is recommended to download the model weights to a shared directory across multiple nodes.
The W8A8 variant (647GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full
A3 node (8 cards, 16 dies) or two A2 nodes.
</Info>
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx
+13 -8
View File
@@ -30,7 +30,7 @@ recommended to use v0.5.13 or a later version.
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 3 4 5 6 7 8` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6 7 8` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -60,16 +60,21 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [DeepSeek-V3.2-W8A8](https://www.modelscope.cn/models/sgl-npu/DeepSeek-V3.2-W8A8) (Quantized version)
- [DeepSeek-V3.2-W8A8](https://www.modelscope.cn/models/sgl-npu/DeepSeek-V3.2-W8A8) (Quantized version, 694.47GB)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
<Info>
The W8A8 variant (694.47GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full
A3 node (8 cards, 16 dies) or two A2 nodes.
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx
+18 -13
View File
@@ -1,7 +1,7 @@
---
title: "GLM-5.1"
metatags:
description: "Deploy GLM-5.1 model with SGLang on Ascend NPUs, including single-node and multi-node deployment modes."
description: "Deploy GLM-5.1 model with SGLang on Ascend NPUs, including multi-node PD mixed and PD disaggregation modes."
---
## Introduction
@@ -13,8 +13,8 @@ The model features built-in bilingual (Chinese-English) capabilities with a unif
reasoning, math, code, and tool calling tasks. GLM-5.1 supports both Thinking mode (step-by-step reasoning) and Instruct
mode (direct response), with a native context window of approximately 200k tokens.
This document demonstrates the deployment of GLM-5.1 on Ascend NPUs using SGLang, including single-node and multi-node
deployment, feature configuration, and performance optimization.
This document demonstrates the deployment of GLM-5.1 on Ascend NPUs using SGLang, including multi-node PD mixed mode,
multi-node PD disaggregation mode, feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.13**. The current model (GLM-5.1) is fully supported in
this version. To use the latest features (e.g., speculative decoding, multi-node deployment), it is recommended to use
@@ -30,8 +30,8 @@ v0.5.13 or a later version.
| Context Parallelism | `--enable-nsa-prefill-context-parallel \`<br/>`--nsa-prefill-cp-mode in-seq-split \`<br/>`--attn-cp-size 4` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 3 4 5 6` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -62,18 +62,23 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [GLM-5.1](https://www.modelscope.cn/models/ZhipuAI/GLM-5.1) (BF16)
- [GLM-5.1-w4a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) (Quantized version)
- [GLM-5.1](https://www.modelscope.cn/models/ZhipuAI/GLM-5.1) (BF16, 1.51TB)
- [GLM-5.1-w4a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) (Quantized version, 420.17GB)
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively.
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
<Info>
We recommend deploying the W4A8 variant for reduced resource usage and higher throughput.
It (420.17GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards).
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx
+18 -78
View File
@@ -1,7 +1,7 @@
---
title: "GLM-5.2"
metatags:
description: "Deploy GLM-5.2 model with SGLang on Ascend NPUs, including single-node and multi-node deployment modes."
description: "Deploy GLM-5.2 model with SGLang on Ascend NPUs, including single-node, multi-node, and PD disaggregation modes."
---
## Introduction
@@ -26,8 +26,8 @@ multi-node deployment, prefill-decode disaggregation, feature configuration, and
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 16` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 16` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -52,24 +52,29 @@ Before following this tutorial, complete the environment setup in the documents
It walks you through launching the official container image, starting the SGLang server, and sending a test request.
Recommended if you are new to SGLang on Ascend.
- [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide.
It covers the component version mapping (CANN, PyTorch adapter, Triton, kernels, etc.), building from source or from a
It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
the environment instead of using the official image.
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [GLM-5.2](https://huggingface.co/collections/zai-org/glm-52) (BF16)
- [GLM-5.2-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/) (Quantized version without MTP)
- [GLM-5.2](https://www.modelscope.cn/models/ZhipuAI/GLM-5.2) (BF16, 1.51TB)
- [GLM-5.2-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/) (Quantized version without MTP, 774.08GB)
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively.
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
<Info>
We recommend deploying the W8A8 variant for reduced resource usage and higher throughput.
It (774.08GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes.
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
@@ -168,10 +173,7 @@ swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.
### Single-node deployment
Quantized model `GLM-5.2-w8a8` can be deployed on one Atlas 800I A3 node or one Atlas 800I A2 node.
<Tabs>
<Tab title="Atlas 800I A3">
Quantized model `GLM-5.2-w8a8` can be deployed on one Atlas 800I A3 node.
Run the following script to execute online inference.
@@ -234,68 +236,6 @@ python3 -m sglang.launch_server \
--moe-a2a-backend deepep --deepep-mode auto
```
</Tab>
<Tab title="Atlas 800I A2">
Run the following script to execute online inference.
```shell
# ============================================================
# Before running, update the following variables:
# MODEL_PATH: path to the model weights directory
# ============================================================
export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export STREAMS_PER_DEVICE=32
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export HCCL_BUFFSIZE=1000
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
export TRANSFORMERS_VERBOSITY=error
#DEEPEP
export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
MODEL_PATH=/path/to/model-weights
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--attention-backend ascend \
--device npu \
--tp-size 8 \
--nnodes 1 \
--dp-size 1 \
--enable-dp-attention \
--chunked-prefill-size -1 \
--max-prefill-tokens 65536 \
--trust-remote-code \
--mem-fraction-static 0.9 \
--served-model-name glm-5 \
--cuda-graph-bs 8 \
--max-running-requests 102 \
--quantization modelslim \
--speculative-draft-model-quantization unquant \
--moe-a2a-backend deepep --deepep-mode auto \
--load-balance-method round_robin
```
</Tab>
</Tabs>
### Multi-node deployment
Quantized model `GLM-5.2-w8a8` can be deployed on two Atlas 800I A3 nodes.
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx
+45 -13
View File
@@ -1,7 +1,7 @@
---
title: "Kimi-K2.6"
metatags:
description: "Deploy Kimi-K2.6 model with SGLang on Ascend NPUs, including single-node deployment modes."
description: "Deploy Kimi-K2.6 model with SGLang on Ascend NPUs, including single-node, multi-node, and PD disaggregation modes."
---
## Introduction
@@ -13,7 +13,8 @@ active parameters. The model seamlessly integrates vision and language understan
capabilities, supporting both instant and thinking modes as well as conversational and agentic paradigms.
This document demonstrates the deployment of Kimi-K2.6 on Ascend NPUs using SGLang, including single-node PD mixed
mode, feature configuration, and performance optimization.
mode, multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance
optimization.
This document is validated and written based on **SGLang v0.5.13**. The current model (Kimi-K2.6) is fully supported in
this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use
@@ -28,8 +29,8 @@ v0.5.13 or a later version.
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 4 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 5 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -62,20 +63,24 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Kimi-K2.6](https://www.modelscope.cn/models/moonshotai/Kimi-K2.6) (BF16)
- [Kimi-K2.6-w4a8](https://www.modelscope.cn/models/Eco-Tech/Kimi-K2.6-w4a8) (W4A8 quantized version)
- [Kimi-K2.6](https://www.modelscope.cn/models/moonshotai/Kimi-K2.6) (BF16, 595.21GB)
- [Kimi-K2.6-w4a8](https://www.modelscope.cn/models/Eco-Tech/Kimi-K2.6-w4a8) (W4A8 quantized version, 535.91GB)
- [kimi-k2.6-eagle3](https://www.modelscope.cn/models/lightseekorg/kimi-k2.6-eagle3) (EAGLE3 draft model for speculative decoding)
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Kimi-K2.6-w4a8` from `Kimi-K2.6`.
You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Kimi-K2.6-w4a8` from `Kimi-K2.6`.
<Info>
We recommend deploying the W4A8 variant for reduced resource usage and higher throughput.
It (535.91GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes.
</Info>
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional cards.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
It is recommended to download the model weights to a shared directory across multiple nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
@@ -208,6 +213,33 @@ curl http://${HOST}:${PORT}/generate \
Expected result: an HTTP 200 response with the generated text containing "Paris".
For multimodal requests (text + image):
```shell
# ============================================================
# Before running, update the following variables:
# HOST: the server host address (e.g., localhost)
# PORT: the server port number (e.g., 6689)
# ============================================================
curl http://${HOST}:${PORT}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Kimi-K2.6-w4a8",
"messages": [
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
{"type": "text", "text": "Describe this image."}
]
}
]
}'
```
Expected result: an HTTP 200 response with a description of the image.
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx
+13 -8
View File
@@ -26,7 +26,7 @@ v0.5.13 or a later version.
| Expert Parallelism | `--moe-a2a-backend deepep \`<br/>`--deepep-mode low_latency` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` |
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--enable-multi-layer-eagle` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -52,22 +52,27 @@ Before following this tutorial, complete the environment setup in the documents
It walks you through launching the official container image, starting the SGLang server, and sending a test request.
Recommended if you are new to SGLang on Ascend.
- [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide.
It covers the component version mapping (CANN, PyTorch adapter, Triton, kernels, etc.), building from source or from a
It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
the environment instead of using the official image.
### Model weights
<Warning>
Before downloading model weights, check the model size to reserve enough disk space.
</Warning>
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [MiMo-V2-Flash-W8A8](https://modelers.cn/models/Modelers_Park/MiMo-V2-Flash-W8A8) (Quantized version)
- [MiMo-V2-Flash-W8A8](https://www.modelscope.cn/models/iridiumine/MiMo-V2-Flash-W8A8) (Quantized version, 311.5GB)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
<Info>
The W8A8 variant (311.5GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2
node or 8 dies on A3 (4 cards).
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx
+16 -13
View File
@@ -1,7 +1,7 @@
---
title: "MiniMax-M2.5"
metatags:
description: "Deploy MiniMax-M2.5 model with SGLang on Ascend NPUs, including single-node and PD disaggregation modes."
description: "Deploy MiniMax-M2.5 model with SGLang on Ascend NPUs, including single-node deployment mode."
---
## Introduction
@@ -13,10 +13,10 @@ custom eagle model for accelerated inference, and excels at general language und
long-context tasks.
This document demonstrates the deployment of MiniMax-M2.5 on Ascend NPUs using SGLang, including single-node PD mixed
mode, multi-node PD disaggregation mode, feature configuration, and performance optimization.
mode, feature configuration, and performance optimization.
This document is validated and written based on **SGLang v0.5.13**. The current model (MiniMax-M2.5) is fully supported in
this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use
this version. To use the latest features (e.g., speculative decoding), it is recommended to use
v0.5.13 or a later version.
## Supported features
@@ -26,10 +26,9 @@ v0.5.13 or a later version.
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 16` |
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend ascend_fuseep \`<br/>`--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 8 16 24 32 48 64 80` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 8 16 24 32 48 64 80` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
@@ -59,17 +58,21 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
- [MiniMax-M2.5-w8a8-QuaRot](https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot) (W8A8 quantized version)
- [MiniMax-M2.5-w8a8-QuaRot](https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot) (W8A8 quantized version, 230.82GB)
- [MiniMax-M2.5-eagle-model](https://www.modelscope.cn/models/sgl-npu/MiniMax-M2.5-eagel-model-0318) (EAGLE3 draft model for speculative decoding)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
<Info>
The W8A8 variant (230.82GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2
node or 8 dies on A3 (4 cards).
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx
+15 -10
View File
@@ -25,8 +25,8 @@ version.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 6 9 10 15 16` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 6 9 10 15 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
@@ -56,18 +56,23 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Qwen3-8B](https://www.modelscope.cn/models/Qwen/Qwen3-8B) (BF16)
- [Qwen3-8B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-8B-w8a8) (W8A8 quantized version)
- [Qwen3-8B](https://www.modelscope.cn/models/Qwen/Qwen3-8B) (BF16, 16.40GB)
- [Qwen3-8B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-8B-w8a8) (W8A8 quantized version, 11.27GB)
- [Eagle3-Qwen3-8B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-8B-zh) (EAGLE3 draft model for speculative decoding)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional cards.
<Info>
We recommend deploying the W8A8 variant for reduced resource usage and higher throughput.
It (11.27GB) fits within a single 64GB die, so `--tp-size 1` is sufficient on either A2 or A3.
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx
+21 -12
View File
@@ -1,7 +1,7 @@
---
title: "Qwen3-235B-A22B"
metatags:
description: "Deploy Qwen3-235B-A22B model with SGLang on Ascend NPUs, including single-node, multi-node, and PD disaggregation modes."
description: "Deploy Qwen3-235B-A22B model with SGLang on Ascend NPUs, including single-node PD mixed and multi-node PD disaggregation modes."
---
## Introduction
@@ -13,8 +13,8 @@ logical reasoning, text comprehension, mathematics, science, coding, and tool us
and thinking/reasoning-enhanced editions.
This document demonstrates the deployment of Qwen3-235B-A22B on Ascend NPUs using SGLang, including single-node PD mixed
mode, multi-node PD mixed mode, multi-node PD disaggregation mode, 256k long-sequence inference, Prefill Context Parallel,
feature configuration, and performance optimization.
mode, multi-node PD disaggregation mode, 256k long-sequence inference, Prefill Context Parallel, feature configuration,
and performance optimization.
This document is validated and written based on **SGLang v0.5.13**. The current model (Qwen3-235B-A22B) is fully supported in
this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use
@@ -29,8 +29,8 @@ v0.5.13 or a later version.
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend ascend_fuseep` |
| PD Disaggregation | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 94208` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 94208` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -62,18 +62,25 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
Before downloading model weights, check the model size to reserve enough disk space.
</Warning>
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Qwen3-235B-A22B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-235B-A22B-Instruct-2507) (BF16)
- [Qwen3-235B-A22B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-235B-A22B-W8A8) (W8A8 quantized version)
- [Qwen3-235B-A22B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-235B-A22B-Instruct-2507) (BF16, 470.21GB)
- [Qwen3-235B-A22B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-235B-A22B-W8A8) (W8A8 quantized version, 236.80GB)
- [Qwen3-235B-A22B-Eagle3](https://www.modelscope.cn/models/nv-community/Qwen3-235B-A22B-Eagle3) (EAGLE3 draft model for speculative decoding)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
<Info>
The BF16 variant (470.21GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full
A3 node (8 cards, 16 dies) or two A2 nodes.
It is recommended to download the model weights to a shared directory across multiple nodes.
The W8A8 variant (236.80GB) can be deployed on 8 × 64GB (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards).
</Info>
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
@@ -218,6 +225,7 @@ python3 -m sglang.launch_server \
--chunked-prefill-size -1 \
--skip-server-warmup \
--device npu \
--quantization modelslim \
--tp-size 16 \
--mem-fraction-static 0.45 \
--max-running-requests 1 \
@@ -250,6 +258,7 @@ python3 -m sglang.launch_server \
--mem-fraction-static 0.8 \
--disable-cuda-graph \
--device npu \
--quantization modelslim \
--disable-radix-cache \
--chunked-prefill-size 8192 \
--skip-server-warmup \
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx
+12 -8
View File
@@ -26,7 +26,7 @@ v0.5.13 or a later version.
| Tensor Parallelism | `--tp-size 2` |
| Data Parallelism | `--dp-size 2` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
@@ -56,18 +56,22 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
Before downloading model weights, check the model size to reserve enough disk space.
</Warning>
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Qwen3-30B-A3B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-30B-A3B-Instruct-2507) (BF16, recommended)
- [Qwen3-30B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8) (W8A8 quantized version)
- [Qwen3-30B-A3B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-30B-A3B-Instruct-2507) (BF16, 61.08GB, recommended)
- [Qwen3-30B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8) (W8A8 quantized version, 31.29GB)
- [Qwen3-a3B_eagle3](https://www.modelscope.cn/models/vllm-ascend/Qwen3-a3B_eagle3) (EAGLE3 draft model for speculative decoding)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional cards.
<Info>
Both variants fit within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3.
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx
+15 -9
View File
@@ -24,7 +24,7 @@ version.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 4` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 4 8 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 4 8 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
@@ -54,18 +54,24 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Qwen3-32B](https://www.modelscope.cn/models/Qwen/Qwen3-32B) (BF16)
- [Qwen3-32B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-32B-W8A8) (W8A8 quantized version)
- [Qwen3-32B](https://www.modelscope.cn/models/Qwen/Qwen3-32B) (BF16, 65.54GB)
- [Qwen3-32B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-32B-W8A8) (W8A8 quantized version, 42.77GB)
- [Eagle3-Qwen3-32B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-32B-zh) (EAGLE3 draft model for speculative decoding)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional cards.
<Info>
The BF16 variant (65.54GB) can be deployed on 2 × 64GB of device memory (`--tp-size 2`), which corresponds to 2 cards on A2 or 1 card (2 dies) on A3.
It is recommended to download the model weights to a shared directory across multiple nodes.
The W8A8 variant (42.77GB) fits within a single die.
</Info>
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx
+40 -8
View File
@@ -28,7 +28,7 @@ use v0.5.13 or a later version.
| Data Parallelism | `--dp-size 8` |
| Expert Parallelism | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -59,16 +59,21 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP)
- [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP, 235.88GB)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
<Info>
The W4A8 variant (235.88GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2
node or 8 dies on A3 (4 cards).
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
@@ -186,6 +191,33 @@ curl http://${HOST}:${PORT}/generate \
Expected result: an HTTP 200 response with the generated text containing "Paris".
For multimodal requests (text + image):
```shell
# ============================================================
# Before running, update the following variables:
# HOST: the server host address (e.g., localhost)
# PORT: the server port number (e.g., 6688)
# ============================================================
curl http://${HOST}:${PORT}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3.5-397B-A17B-w4a8-mtp",
"messages": [
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
{"type": "text", "text": "Describe this image."}
]
}
]
}'
```
Expected result: an HTTP 200 response with a description of the image.
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx
+40 -9
View File
@@ -25,8 +25,8 @@ v0.5.13 or a later version.
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 8 16 32 48` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 8 16 32 48` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
@@ -56,16 +56,20 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Eco-Tech/Qwen3.6-27B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) (W8A8 quantized version)
- [Eco-Tech/Qwen3.6-27B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) (W8A8 quantized version, 36.45GB)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional cards.
<Info>
The W8A8 variant (36.45GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3.
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
@@ -183,6 +187,33 @@ curl http://${HOST}:${PORT}/generate \
Expected result: an HTTP 200 response with the generated text containing "Paris".
For multimodal requests (text + image):
```shell
# ============================================================
# Before running, update the following variables:
# HOST: the server host address (e.g., localhost)
# PORT: the server port number (e.g., 6688)
# ============================================================
curl http://${HOST}:${PORT}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3.6-27B-w8a8",
"messages": [
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
{"type": "text", "text": "Describe this image."}
]
}
]
}'
```
Expected result: an HTTP 200 response with a description of the image.
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx
+40 -9
View File
@@ -25,8 +25,8 @@ v0.5.13 or a later version.
| Feature | Example usage |
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 2` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
@@ -56,16 +56,20 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Eco-Tech/Qwen3.6-35B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) (W8A8 quantized version)
- [Eco-Tech/Qwen3.6-35B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) (W8A8 quantized version, 39.81GB)
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional cards.
<Info>
The W8A8 variant (39.81GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3.
</Info>
It is recommended to download the model weights to a shared directory across multiple nodes.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation
@@ -183,6 +187,33 @@ curl http://${HOST}:${PORT}/generate \
Expected result: an HTTP 200 response with the generated text containing "Paris".
For multimodal requests (text + image):
```shell
# ============================================================
# Before running, update the following variables:
# HOST: the server host address (e.g., localhost)
# PORT: the server port number (e.g., 6688)
# ============================================================
curl http://${HOST}:${PORT}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3.6-35B-A3B-w8a8",
"messages": [
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
{"type": "text", "text": "Describe this image."}
]
}
]
}'
```
Expected result: an HTTP 200 response with a description of the image.
Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).
docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx
+14 -12
View File
@@ -27,7 +27,7 @@ v0.5.13 or a later version.
| Data Parallelism | `--dp-size 2` |
| Expert Parallelism | `--ep-size 4 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 4 8` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 8` |
| Speculative Decoding | `--speculative-algorithm NEXTN \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--speculative-draft-model-quantization unquant \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
@@ -58,21 +58,23 @@ Before following this tutorial, complete the environment setup in the documents
### Model weights
<Warning>
If you need to download model weights, check the model size before downloading to reserve enough space.
</Warning>
Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.
- [Qwen3-Next-80B-A3B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen3-Next-80B-A3B-Instruct) (BF16)
- [Qwen3-Next-80B-A3B-Instruct-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-Next-80B-A3B-Instruct-W8A8) (W8A8 quantized version)
- [Qwen3-Next-80B-A3B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen3-Next-80B-A3B-Instruct) (BF16, 162.68GB) — used as the EAGLE3 draft model
- [Qwen3-Next-80B-A3B-Instruct-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-Next-80B-A3B-Instruct-W8A8) (W8A8 quantized version, 84.90GB)
<Note>
For EAGLE3 speculative decoding, use the BF16 weights as the draft model (set `--speculative-draft-model-path` to the BF16 weight path).
</Note>
<Info>
We recommend deploying the W8A8 variant as the main model for reduced resource usage and higher throughput.
It (84.90GB) can be deployed on 2 × 64GB (`--tp-size 2`), which corresponds to 2 cards on A2 or 1 card (2 dies) on A3.
The BF16 weights serve as the EAGLE3 draft model (set `--speculative-draft-model-path` to the BF16 weight path).
</Info>
Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency,
refer to the [best practice configurations](#best-practices) which may require additional nodes or cards.
This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.
It is recommended to download the model weights to a shared directory across multiple nodes.
For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
## Installation