diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx index c8fed3298..89d8197fc 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx @@ -107,7 +107,7 @@ conda config --set show_channel_urls yes conda config --remove channels defaults ``` -Edit the system-level conda config to remove any hardcoded defaults, e.g. vi ~/miniconda3/.condarc +Edit the system-level conda config to remove any hardcoded defaults, e.g., vi ~/miniconda3/.condarc Then remove the failed environment and recreate it: ```bash Command @@ -140,7 +140,7 @@ installed in addition to MemFabric-Hybrid. pip install memfabric-zbal==1.1.1 ``` -#### Pytorch and Pytorch Framework Adaptor on Ascend +#### PyTorch and PyTorch Framework Adaptor on Ascend ```bash Command PYTORCH_VERSION=2.10.0 @@ -242,7 +242,7 @@ git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image -# Replace with the target architecture, e.g. amd64, arm64. +# Replace with the target architecture, e.g., amd64, arm64. # Optional build arguments: # --build-arg DEVICE_TYPE=910b # Required for Atlas 800I A2 # --build-arg APTMIRROR= # Use a custom APT mirror to improve download speed diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx index 2c7fc9088..77f2faa59 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx @@ -23,7 +23,7 @@ on Ascend hardware. ## Directory Structure -The identifiers `sgl_kenel_npu_ops.h`, `KernalHelloworld`, and `retrive_*` (e.g. `retrive_index`, +The identifiers `sgl_kenel_npu_ops.h`, `KernalHelloworld`, and `retrive_*` (e.g., `retrive_index`, `retrive_next_token`, `retrive_next_sibling`) in this guide match the spelling used in the upstream [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository and are kept verbatim for consistency. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx index 215d1ea7e..f50b01331 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx @@ -745,7 +745,11 @@ Separates the prefill (P) and decode (D) phases onto different device groups, al ### Quantization (`--quantization`) -Reduces model weight and activation precision (e.g., W8A8, W4A8, W4A16) to decrease memory usage and increase throughput. On Ascend, the natively supported and highly optimized quantization method is ModelSlim (supporting W4A4, W8A8, W4A8 dynamic/static). Support for other community formats (such as AWQ, GPTQ, Auto-round, and Compressed-tensors) depends on the availability of specific Ascend custom kernels in your environment; see quantization for details and compatibility matrices. +Reduces model weight and activation precision (e.g., W8A8, W4A8, W4A16) to decrease memory usage and increase throughput. +On Ascend, the natively supported and highly optimized quantization method is ModelSlim (supporting W4A4, W8A8, W4A8 dynamic/static). +Support for other community formats (such as AWQ, GPTQ, Auto-round, and Compressed-tensors) depends on the availability +of specific Ascend custom kernels in your environment; +see [Quantization on Ascend](/docs/hardware-platforms/ascend-npus/ascend_npu_quantization) for details and compatibility matrices. ### Chunked Prefill (`--chunked-prefill-size`) diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx index d25470405..6fdf608f3 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx @@ -221,7 +221,7 @@ sending `/stop_profile`. `bench_serving --profile` creates a timestamp subdirectory inside -`--profile-output-dir` (e.g. `//`). The output path is +`--profile-output-dir` (e.g., `//`). The output path is shown in the server log as `Profiling done. Traces are saved to: `. @@ -393,7 +393,7 @@ set of supported parameters: profile_stages - Stages to profile, e.g. ["prefill", "decode"]. + Stages to profile, e.g., ["prefill", "decode"]. Requires profile_by_stage None diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx index cd8ab03fe..ee1b63a63 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx @@ -307,7 +307,7 @@ Compressed-tensors (LLM Compressor) on Ascend support: **Usage Examples:** -- Dense model (e.g. Qwen3-14B-Q4_K_M.gguf): +- Dense model (e.g., Qwen3-14B-Q4_K_M.gguf): ```bash Command python3 -m sglang.launch_server \ @@ -317,7 +317,7 @@ python3 -m sglang.launch_server \ --mem-fraction-static 0.7 --tp-size 2 ``` -- MoE model (e.g. Qwen3-30B-A3B-Q4_K_M.gguf): +- MoE model (e.g., Qwen3-30B-A3B-Q4_K_M.gguf): ```bash Command python3 -m sglang.launch_server \ @@ -333,7 +333,7 @@ python3 -m sglang.launch_server \ > - MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation. > - TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models. -**MXFP8 for LLM dense models (e.g. Qwen3 / Qwen3.5):** +**MXFP8 for LLM dense models (e.g., Qwen3 / Qwen3.5):** LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend A5 series or newer (`npu_dynamic_mx_quant` is not available on A2 / A3). @@ -363,7 +363,7 @@ python3 -m sglang.launch_server \ > - Offline path: `ModelSlimMXFP8Scheme` loads `float8_e4m3fn` weights + `float8_e8m0fnu` block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (`.data` assignment) — calling `.contiguous()` would physically reorder the pre-quantized layout and break the block-scale mapping. > - MoE MXFP8 (FusedMoE/TP) for LLMs is tracked separately and not part of this PR. -**MXFP4 W4A8 for LLM dense models (e.g. Qwen3 / Qwen3.5):** +**MXFP4 W4A8 for LLM dense models (e.g., Qwen3 / Qwen3.5):** LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend A5 series or newer. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx index 7908cf5d6..63fc7b065 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx @@ -17,25 +17,43 @@ see [SGLang installation with NPUs support](/docs/hardware-platforms/ascend-npus - Atlas 800I A2 inference series (Atlas 800I A2) - Atlas 800I A3 inference series (Atlas 800I A3) +To identify your device, run `npu-smi info -l`: A3 reports `Chip Count: 2` per NPU, while A2 reports `Chip Count: 1` per NPU. +For hardware details, see the [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). + +### Docker + +Ensure Docker is installed and the Docker daemon is running on your host machine. Verify with: + +```bash +docker --version && docker info +``` + +If Docker is not installed, follow the [official Docker installation guide](https://docs.docker.com/engine/install/) for +your operating system. + ## Setup environment using container -Ensure sufficient disk space before proceeding. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. +Ensure sufficient disk space before proceeding. Run `df -h` to check the available disk space. The Docker image requires +at least **30GB** of free space. If you need to download model weights, check the model size at +[ModelScope](https://www.modelscope.cn/models) to reserve enough space. -We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.13.post1-cann9.0.0-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes. +We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.13.post1-cann9.0.0-a3`) +if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes. +If you have already downloaded model weights to a local path (e.g., `/path/to/model`), mount the path into the container by +adding `--volume /path/to/model:/path/to/model` to the `docker run` command below. + ```shell Command -# Stable release -export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3 - -# Daily build -export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3 +# Choose one (uncomment the line you want): +export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3 # Stable release +# export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3 # Daily build docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ @@ -58,11 +76,9 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ ```shell Command -# Stable release -export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b - -# Daily build -export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b +# Choose one (uncomment the line you want): +export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b # Stable release +# export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b # Daily build docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ @@ -88,7 +104,8 @@ The SGLang server is installed in the container by default. You can use `pip sho ### Start SGLang server -SGLang will automatically download the model from Hugging Face. +SGLang will automatically download the model from Hugging Face. If the model is already downloaded to a local path +(and has been mounted into the container), use that path directly like `--model-path /path/to/model`. ```shell Command # Set HF_ENDPOINT to a mirror site if network is not available @@ -102,7 +119,7 @@ export HF_TOKEN= sglang serve --model-path Qwen/Qwen2.5-7B-Instruct --attention-backend ascend & ``` -If you see output like the following, the server is running. +Server startup may take several minutes. Once you see output like the following, the server is running. ```log Output INFO: Waiting for application startup. @@ -138,13 +155,14 @@ SGLANG_PID=$(pgrep -f "sglang serve") kill -SIGINT $SGLANG_PID ``` -The output should be like the following: +Wait a moment for the server to shut down gracefully. The output should be like the following: ```log Output INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. -INFO: Finished server process [25310] +INFO: Finished server process [] ``` -The server has now stopped. You can verify it with `ps -ef | grep sglang`, then exit the container by pressing `Ctrl+D`. +The server has now stopped. You can verify it with `ps -ef | grep sglang` — the expected output is nothing (no matching process), +then exit the container by pressing `Ctrl+D`. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_reference.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_reference.mdx index 667be986f..31e482037 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_reference.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_reference.mdx @@ -87,7 +87,7 @@ PD disaggregation separates inference into two stages running on different nodes | **1P2D** | 1 prefill node + 2 decode nodes | You will see these in [Best Practice](/docs/hardware-platforms/ascend-npus/best_practice) -section headings, e.g. `W8A8 2P1D 32P` means "W8A8 quantization, 2 prefill nodes + +section headings, e.g., `W8A8 2P1D 32P` means "W8A8 quantization, 2 prefill nodes + 1 decode node, 32 cards total." ### Parallelism strategies diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx index 513ccd98b..87cb24fc5 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx @@ -3,7 +3,7 @@ title: "Support Features on Ascend NPU" metatags: description: "Documentation for Support Features on Ascend NPU" --- -This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any +This section describes the basic functions and features supported by the Ascend NPU. If you encounter issues or have any questions, please [open an issue](https://github.com/sgl-project/sglang/issues). If you want to know the meaning and usage of each parameter, @@ -279,7 +279,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--quantization-param-path` `None` Type: str - Special For GPU + Special for GPU `--kv-cache-dtype` @@ -297,37 +297,37 @@ click [Server Arguments](../../advanced_features/server_arguments). `--modelopt-quant` `None` Type: str - Special For GPU + Special for GPU `--modelopt-checkpoint-restore-path` `None` Type: str - Special For GPU + Special for GPU `--modelopt-checkpoint-save-path` `None` Type: str - Special For GPU + Special for GPU `--modelopt-export-path` `None` Type: str - Special For GPU + Special for GPU `--quantize-and-serve` `False` bool flag
(set to enable) - Special For GPU + Special for GPU `--rl-quant-profile` `None` Type: str - Special For GPU + Special for GPU @@ -1157,7 +1157,7 @@ click [Server Arguments](../../advanced_features/server_arguments). -## LoRA(Restrict the use Qwen Series Models) +## LoRA (restricted to Qwen series models) @@ -1180,13 +1180,13 @@ click [Server Arguments](../../advanced_features/server_arguments). - + - + @@ -1227,8 +1227,8 @@ click [Server Arguments](../../advanced_features/server_arguments). - - + + @@ -1264,7 +1264,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
`--enable-lora` `False`Bool flag
(set to enable)		bool flag
(set to enable)		 A2, A3
`--enable-lora-overlap-loading` `False`Bool flag
(set to enable)		bool flag
(set to enable)		 A2, A3
`--lora-backend``ascend``ascend``csgmv``triton`,
`csgmv`,
`ascend`,
`torch_native`		 A2, A3
-## Kernel Backends (Attention, Sampling, Grammar, GEMM) +## Kernel backends (attention, sampling, grammar, GEMM) @@ -1741,7 +1741,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
-## Mamba Cache +## Mamba cache @@ -2544,7 +2544,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD - + @@ -2608,7 +2608,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD - + @@ -2733,7 +2733,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
`--disaggregation-transfer-backend` `mooncake``ascend``ascend` (default `mooncake` not supported on NPU, must be specified manually) A2, A3
`--enable-adaptive-dispatch-to-encoder` `False`bool flag
(set to enable adaptively dispatch)		bool flag
(set to enable adaptive dispatch)		 A2, A3
-## For PD-Multiplexing +## For PD-multiplexing @@ -2774,7 +2774,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
-## For Multi-Modal +## For multi-modal @@ -2951,10 +2951,10 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
-## Other Params +## Other params -The following parameters are not supported because the third-party components that depend on are not compatible with the -NPU, like Ktransformer, checkpoint-engine etc. +The following parameters are not supported because the required third-party components +(e.g., Ktransformer, checkpoint-engine) are not compatible with the NPU. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx index a0d61b5b5..096871180 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx @@ -608,7 +608,7 @@ You are welcome to enable various models based on your business requirements. - + diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx index b45f8b841..dcc44e2ec 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx @@ -64,7 +64,10 @@ Please note all your testing and benchmarking results in PR description. ### Benchmark -- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow [https://docs.sglang.io/developer_guide/benchmark_and_profiling.html](https://docs.sglang.io/developer_guide/benchmark_and_profiling.html) to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). +- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) + to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than + that from HF Transformer run. Similarly, follow the [benchmark and profiling guide](/docs/developer_guide/benchmark_and_profiling) + to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). - **(Optional) Other evals**: If you ran other evals, please note the results in PR description. @@ -99,7 +102,8 @@ To port a model from vLLM to SGLang: patterns, such as replacing CUDA kernels with `torch_npu` equivalents. The NPU backend is at `sglang/srt/hardware_backend/npu/`. -Note: make sure you add your new model to the supported models list in the supported models documentation. +Note: make sure you add your new model to the supported models list in the +[supported models documentation](https://github.com/sgl-project/sglang/blob/main/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx). ## Registering an External Model Implementation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx index b393cdaef..64f5cb186 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx @@ -538,7 +538,7 @@ python -m sglang_router.launch_router \ We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 115200` = `int(128000 * 0.9)` is the shared prefix portion. -`--gsp-question-len 12799` = `int(128000 * (1 - 0.9))` is the unique per-request suffix. +`--gsp-question-len 12800` = `int(128000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command @@ -550,7 +550,7 @@ python -m sglang.bench_serving \ --gsp-num-groups 1 \ --gsp-prompts-per-group 4 \ --gsp-system-prompt-len 115200 \ - --gsp-question-len 12799 \ + --gsp-question-len 12800 \ --gsp-output-len 1000 \ --max-concurrency 1 \ --num-prompts 4 \ @@ -931,7 +931,7 @@ python -m sglang_router.launch_router \ We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion. -`--gsp-question-len 6399` = `int(64000 * (1 - 0.9))` is the unique per-request suffix. +`--gsp-question-len 6400` = `int(64000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command @@ -943,7 +943,7 @@ python -m sglang.bench_serving \ --gsp-num-groups 1 \ --gsp-prompts-per-group 8 \ --gsp-system-prompt-len 57600 \ - --gsp-question-len 6399 \ + --gsp-question-len 6400 \ --gsp-output-len 1500 \ --max-concurrency 2 \ --num-prompts 8 \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx index 687f09e3a..0d8c669c1 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx @@ -16,7 +16,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1080p-30-out256-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-2p-in64k-out1k-prefix90-50ms) | +| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in64k-out1k-prefix90-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 128k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) | @@ -224,13 +224,13 @@ python -m sglang.bench_serving \ --random-range-ratio 1 ``` -### Qwen3.6-27B 2P IN64K OUT1K PREFIX90 50ms +### Qwen3.6-27B 1P IN64K OUT1K PREFIX90 50ms **Model**: Qwen3.6-27B **Hardware**: Atlas 800I A3 -**Cards**: 2 +**Cards**: 1 **Deploy Mode**: PD Mixed @@ -307,7 +307,7 @@ python3 -m sglang.launch_server \ We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion. -`--gsp-question-len 6399` = `int(64000 * (1 - 0.9))` is the unique per-request suffix. +`--gsp-question-len 6400` = `int(64000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command @@ -319,7 +319,7 @@ python -m sglang.bench_serving \ --gsp-num-groups 1 \ --gsp-prompts-per-group 80 \ --gsp-system-prompt-len 57600 \ - --gsp-question-len 6399 \ + --gsp-question-len 6400 \ --gsp-output-len 1000 \ --max-concurrency 20 \ --num-prompts 80 \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx index d21063828..07e430428 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx @@ -23,7 +23,6 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) | @@ -327,108 +326,6 @@ python -m sglang.bench_serving \ --random-range-ratio 1 ``` -### Qwen3.6-35B-A3B 1P IN128K OUT1K PREFIX90 50ms - -**Model**: Qwen3.6-35B-A3B - -**Hardware**: Atlas 800I A3 - -**Cards**: 1 - -**Deploy Mode**: PD Mixed - -**Quantization**: BF16 - -**Dataset**: 128k+1k (90% prefix cache hit rate) - -**TPOT**: 50ms - -#### Model Deployment - -```bash Command -# ============================================================ -# Before running, update the following variables: -# MODEL_PATH: path to the model weights directory -# HCCL_SOCKET_IFNAME: network interface name for HCCL -# GLOO_SOCKET_IFNAME: network interface name for Gloo -# ============================================================ - -MODEL_PATH=/path/to/model-weights -echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor -sysctl -w vm.swappiness=0 -sysctl -w kernel.numa_balancing=0 -sysctl -w kernel.sched_migration_cost_ns=50000 - -unset https_proxy -unset http_proxy -unset HTTPS_PROXY -unset HTTP_PROXY -unset ASCEND_LAUNCH_BLOCKING - -source /usr/local/Ascend/ascend-toolkit/set_env.sh -source /usr/local/Ascend/nnal/atb/set_env.sh - -export ASCEND_USE_FIA=1 -export GLOO_SOCKET_IFNAME= -export HCCL_OP_EXPANSION_MODE=AIV -export HCCL_SOCKET_IFNAME= -export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True -export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30 -export SGLANG_SET_CPU_AFFINITY=1 -export STREAMS_PER_DEVICE=32 - -python3 -m sglang.launch_server \ - --model-path $MODEL_PATH \ - --host 127.0.0.1 --port 6688 \ - --tp-size 2 \ - --nnodes 1 \ - --attention-backend ascend \ - --device npu \ - --chunked-prefill-size 16384 \ - --max-prefill-tokens 65536 \ - --trust-remote-code \ - --enable-prefill-delayer \ - --mamba-radix-cache-strategy extra_buffer \ - --max-running-requests 103 \ - --max-mamba-cache-size 85 \ - --mem-fraction-static 0.85 \ - --cuda-graph-bs 2 4 8 16 32 48 64 80 96 103 \ - --enable-multimodal \ - --mm-attention-backend ascend_attn \ - --dtype bfloat16 \ - --mamba-ssm-dtype bfloat16 \ - --speculative-algorithm NEXTN \ - --speculative-num-steps 3 \ - --speculative-eagle-topk 1 \ - --speculative-num-draft-tokens 4 \ - --reasoning-parser qwen3 \ - --tool-call-parser qwen3_coder -``` - -#### Benchmark - -We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): -`--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion. -`--gsp-question-len 6399` = `int(64000 * (1 - 0.9))` is the unique per-request suffix. -`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. - -```bash Command -python -m sglang.bench_serving \ - --dataset-name generated-shared-prefix \ - --backend sglang \ - --host 127.0.0.1 \ - --port 6688 \ - --gsp-num-groups 1 \ - --gsp-prompts-per-group 412 \ - --gsp-system-prompt-len 57600 \ - --gsp-question-len 6399 \ - --gsp-output-len 1000 \ - --max-concurrency 103 \ - --num-prompts 412 \ - --request-rate inf -``` - ### Qwen3.6-35B-A3B 1P IN254K OUT1K **Model**: Qwen3.6-35B-A3B diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx index be0b285a2..780a34cfe 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx @@ -14,7 +14,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-20ms) | +| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-1p-in3k5-out1k5-20ms) | | Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 6k+1.5k | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) | ### High Throughput @@ -27,13 +27,13 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend -### Qwen3-Next-80B-A3B-Instruct W8A8 2P IN3K5 OUT1K5 20ms +### Qwen3-Next-80B-A3B-Instruct W8A8 1P IN3K5 OUT1K5 20ms **Model**: Qwen3-Next-80B-A3B-Instruct **Hardware**: Atlas 800I A3 -**Cards**: 2 +**Cards**: 1 **Deploy Mode**: PD Mixed diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx index 1cb391329..6e0e9b6d3 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx @@ -1,7 +1,7 @@ --- title: "DeepSeek-R1" metatags: - description: "Deploy DeepSeek-R1 model with SGLang on Ascend NPUs, including single-node and multi-node PD disaggregation modes." + description: "Deploy DeepSeek-R1 model with SGLang on Ascend NPUs, including single-node PD mixed and multi-node PD disaggregation modes." --- ## Introduction @@ -27,7 +27,7 @@ v0.5.13 or a later version. | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 4 8 20 21 22` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 20 21 22` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -61,17 +61,25 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - Before downloading model weights, check the model size to reserve enough disk space. - +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [DeepSeek-R1-0528-W4A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8) (Quantized version) -- [DeepSeek-R1-0528-W8A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w8a8) (Quantized version) +- [DeepSeek-R1-0528-W4A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8) (Quantized version, 376GB) +- [DeepSeek-R1-0528-W8A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w8a8) (Quantized version, 647GB) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +The W4A8 variant (376GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 +node or 8 dies on A3 (4 cards). -It is recommended to download the model weights to a shared directory across multiple nodes. +The W8A8 variant (647GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full +A3 node (8 cards, 16 dies) or two A2 nodes. + + +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx index adaf95c09..a5b34a8d8 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx @@ -30,7 +30,7 @@ recommended to use v0.5.13 or a later version. | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 3 4 5 6 7 8` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6 7 8` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -60,16 +60,21 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [DeepSeek-V3.2-W8A8](https://www.modelscope.cn/models/sgl-npu/DeepSeek-V3.2-W8A8) (Quantized version) +- [DeepSeek-V3.2-W8A8](https://www.modelscope.cn/models/sgl-npu/DeepSeek-V3.2-W8A8) (Quantized version, 694.47GB) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +The W8A8 variant (694.47GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full +A3 node (8 cards, 16 dies) or two A2 nodes. + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx index 9e132cf07..869498568 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx @@ -1,7 +1,7 @@ --- title: "GLM-5.1" metatags: - description: "Deploy GLM-5.1 model with SGLang on Ascend NPUs, including single-node and multi-node deployment modes." + description: "Deploy GLM-5.1 model with SGLang on Ascend NPUs, including multi-node PD mixed and PD disaggregation modes." --- ## Introduction @@ -13,8 +13,8 @@ The model features built-in bilingual (Chinese-English) capabilities with a unif reasoning, math, code, and tool calling tasks. GLM-5.1 supports both Thinking mode (step-by-step reasoning) and Instruct mode (direct response), with a native context window of approximately 200k tokens. -This document demonstrates the deployment of GLM-5.1 on Ascend NPUs using SGLang, including single-node and multi-node -deployment, feature configuration, and performance optimization. +This document demonstrates the deployment of GLM-5.1 on Ascend NPUs using SGLang, including multi-node PD mixed mode, +multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.13**. The current model (GLM-5.1) is fully supported in this version. To use the latest features (e.g., speculative decoding, multi-node deployment), it is recommended to use @@ -30,8 +30,8 @@ v0.5.13 or a later version. | Context Parallelism | `--enable-nsa-prefill-context-parallel \`
`--nsa-prefill-cp-mode in-seq-split \`
`--attn-cp-size 4` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 3 4 5 6` | +| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -62,18 +62,23 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [GLM-5.1](https://www.modelscope.cn/models/ZhipuAI/GLM-5.1) (BF16) -- [GLM-5.1-w4a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) (Quantized version) +- [GLM-5.1](https://www.modelscope.cn/models/ZhipuAI/GLM-5.1) (BF16, 1.51TB) +- [GLM-5.1-w4a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) (Quantized version, 420.17GB) - You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively. -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. +It (420.17GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx index 1ec51f76b..4864530d8 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx @@ -1,7 +1,7 @@ --- title: "GLM-5.2" metatags: - description: "Deploy GLM-5.2 model with SGLang on Ascend NPUs, including single-node and multi-node deployment modes." + description: "Deploy GLM-5.2 model with SGLang on Ascend NPUs, including single-node, multi-node, and PD disaggregation modes." --- ## Introduction @@ -26,8 +26,8 @@ multi-node deployment, prefill-decode disaggregation, feature configuration, and | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 16` | +| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 16` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -52,24 +52,29 @@ Before following this tutorial, complete the environment setup in the documents It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. - [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide. - It covers the component version mapping (CANN, PyTorch adapter, Triton, kernels, etc.), building from source or from a + It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [GLM-5.2](https://huggingface.co/collections/zai-org/glm-52) (BF16) -- [GLM-5.2-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/) (Quantized version without MTP) +- [GLM-5.2](https://www.modelscope.cn/models/ZhipuAI/GLM-5.2) (BF16, 1.51TB) +- [GLM-5.2-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/) (Quantized version without MTP, 774.08GB) - You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively. -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +We recommend deploying the W8A8 variant for reduced resource usage and higher throughput. +It (774.08GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation @@ -168,10 +173,7 @@ swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9. ### Single-node deployment -Quantized model `GLM-5.2-w8a8` can be deployed on one Atlas 800I A3 node or one Atlas 800I A2 node. - - - +Quantized model `GLM-5.2-w8a8` can be deployed on one Atlas 800I A3 node. Run the following script to execute online inference. @@ -234,68 +236,6 @@ python3 -m sglang.launch_server \ --moe-a2a-backend deepep --deepep-mode auto ``` - - - -Run the following script to execute online inference. - -```shell -# ============================================================ -# Before running, update the following variables: -# MODEL_PATH: path to the model weights directory -# ============================================================ - -export SGLANG_SET_CPU_AFFINITY=1 - -unset https_proxy -unset http_proxy -unset HTTPS_PROXY -unset HTTP_PROXY -unset ASCEND_LAUNCH_BLOCKING -# cann -source /usr/local/Ascend/ascend-toolkit/set_env.sh -source /usr/local/Ascend/nnal/atb/set_env.sh - -export STREAMS_PER_DEVICE=32 -export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 - -export HCCL_BUFFSIZE=1000 -export HCCL_SOCKET_IFNAME=lo -export GLOO_SOCKET_IFNAME=lo -export TRANSFORMERS_VERBOSITY=error - -#DEEPEP -export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 -export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 -export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 -export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 - -MODEL_PATH=/path/to/model-weights - -python3 -m sglang.launch_server \ - --model-path $MODEL_PATH \ - --attention-backend ascend \ - --device npu \ - --tp-size 8 \ - --nnodes 1 \ - --dp-size 1 \ - --enable-dp-attention \ - --chunked-prefill-size -1 \ - --max-prefill-tokens 65536 \ - --trust-remote-code \ - --mem-fraction-static 0.9 \ - --served-model-name glm-5 \ - --cuda-graph-bs 8 \ - --max-running-requests 102 \ - --quantization modelslim \ - --speculative-draft-model-quantization unquant \ - --moe-a2a-backend deepep --deepep-mode auto \ - --load-balance-method round_robin -``` - - - - ### Multi-node deployment Quantized model `GLM-5.2-w8a8` can be deployed on two Atlas 800I A3 nodes. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx index 603fdcb44..6039bd732 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx @@ -1,7 +1,7 @@ --- title: "Kimi-K2.6" metatags: - description: "Deploy Kimi-K2.6 model with SGLang on Ascend NPUs, including single-node deployment modes." + description: "Deploy Kimi-K2.6 model with SGLang on Ascend NPUs, including single-node, multi-node, and PD disaggregation modes." --- ## Introduction @@ -13,7 +13,8 @@ active parameters. The model seamlessly integrates vision and language understan capabilities, supporting both instant and thinking modes as well as conversational and agentic paradigms. This document demonstrates the deployment of Kimi-K2.6 on Ascend NPUs using SGLang, including single-node PD mixed -mode, feature configuration, and performance optimization. +mode, multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance +optimization. This document is validated and written based on **SGLang v0.5.13**. The current model (Kimi-K2.6) is fully supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use @@ -28,8 +29,8 @@ v0.5.13 or a later version. | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 32768` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` | +| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 4 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 5 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -62,20 +63,24 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Kimi-K2.6](https://www.modelscope.cn/models/moonshotai/Kimi-K2.6) (BF16) -- [Kimi-K2.6-w4a8](https://www.modelscope.cn/models/Eco-Tech/Kimi-K2.6-w4a8) (W4A8 quantized version) +- [Kimi-K2.6](https://www.modelscope.cn/models/moonshotai/Kimi-K2.6) (BF16, 595.21GB) +- [Kimi-K2.6-w4a8](https://www.modelscope.cn/models/Eco-Tech/Kimi-K2.6-w4a8) (W4A8 quantized version, 535.91GB) - [kimi-k2.6-eagle3](https://www.modelscope.cn/models/lightseekorg/kimi-k2.6-eagle3) (EAGLE3 draft model for speculative decoding) +- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Kimi-K2.6-w4a8` from `Kimi-K2.6`. -You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Kimi-K2.6-w4a8` from `Kimi-K2.6`. + +We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. +It (535.91GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. + -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional cards. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. -It is recommended to download the model weights to a shared directory across multiple nodes. +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation @@ -208,6 +213,33 @@ curl http://${HOST}:${PORT}/generate \ Expected result: an HTTP 200 response with the generated text containing "Paris". +For multimodal requests (text + image): + +```shell +# ============================================================ +# Before running, update the following variables: +# HOST: the server host address (e.g., localhost) +# PORT: the server port number (e.g., 6689) +# ============================================================ + +curl http://${HOST}:${PORT}/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Kimi-K2.6-w4a8", + "messages": [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, + {"type": "text", "text": "Describe this image."} + ] + } + ] + }' +``` + +Expected result: an HTTP 200 response with a description of the image. + Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx index f96fb1599..516e89bc1 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx @@ -26,7 +26,7 @@ v0.5.13 or a later version. | Expert Parallelism | `--moe-a2a-backend deepep \`
`--deepep-mode low_latency` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--enable-multi-layer-eagle` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` | | DP LM Head | `--enable-dp-lm-head` | @@ -52,22 +52,27 @@ Before following this tutorial, complete the environment setup in the documents It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. - [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide. - It covers the component version mapping (CANN, PyTorch adapter, Triton, kernels, etc.), building from source or from a + It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights - Before downloading model weights, check the model size to reserve enough disk space. - +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [MiMo-V2-Flash-W8A8](https://modelers.cn/models/Modelers_Park/MiMo-V2-Flash-W8A8) (Quantized version) +- [MiMo-V2-Flash-W8A8](https://www.modelscope.cn/models/iridiumine/MiMo-V2-Flash-W8A8) (Quantized version, 311.5GB) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +The W8A8 variant (311.5GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 +node or 8 dies on A3 (4 cards). + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx index c48513f29..b0f8ffc66 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx @@ -1,7 +1,7 @@ --- title: "MiniMax-M2.5" metatags: - description: "Deploy MiniMax-M2.5 model with SGLang on Ascend NPUs, including single-node and PD disaggregation modes." + description: "Deploy MiniMax-M2.5 model with SGLang on Ascend NPUs, including single-node deployment mode." --- ## Introduction @@ -13,10 +13,10 @@ custom eagle model for accelerated inference, and excels at general language und long-context tasks. This document demonstrates the deployment of MiniMax-M2.5 on Ascend NPUs using SGLang, including single-node PD mixed -mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. +mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.13**. The current model (MiniMax-M2.5) is fully supported in -this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use +this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.13 or a later version. ## Supported features @@ -26,10 +26,9 @@ v0.5.13 or a later version. | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend ascend_fuseep \`
`--deepep-mode auto` | -| PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 8192` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 8 16 24 32 48 64 80` | +| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 8 16 24 32 48 64 80` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -59,17 +58,21 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. -- [MiniMax-M2.5-w8a8-QuaRot](https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot) (W8A8 quantized version) +- [MiniMax-M2.5-w8a8-QuaRot](https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot) (W8A8 quantized version, 230.82GB) - [MiniMax-M2.5-eagle-model](https://www.modelscope.cn/models/sgl-npu/MiniMax-M2.5-eagel-model-0318) (EAGLE3 draft model for speculative decoding) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +The W8A8 variant (230.82GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 +node or 8 dies on A3 (4 cards). + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx index a9d4d2b67..9d71f311e 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx @@ -25,8 +25,8 @@ version. |-------------------------------|-----------------------------------------------------------------------------------------------| | Tensor Parallelism | `--tp-size 2` | | Quantization | `--quantization modelslim` | -| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 8192` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 6 9 10 15 16` | +| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 6 9 10 15 16` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -56,18 +56,23 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Qwen3-8B](https://www.modelscope.cn/models/Qwen/Qwen3-8B) (BF16) -- [Qwen3-8B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-8B-w8a8) (W8A8 quantized version) +- [Qwen3-8B](https://www.modelscope.cn/models/Qwen/Qwen3-8B) (BF16, 16.40GB) +- [Qwen3-8B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-8B-w8a8) (W8A8 quantized version, 11.27GB) - [Eagle3-Qwen3-8B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-8B-zh) (EAGLE3 draft model for speculative decoding) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional cards. + +We recommend deploying the W8A8 variant for reduced resource usage and higher throughput. +It (11.27GB) fits within a single 64GB die, so `--tp-size 1` is sufficient on either A2 or A3. + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx index 0462a3b23..871d1704f 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx @@ -1,7 +1,7 @@ --- title: "Qwen3-235B-A22B" metatags: - description: "Deploy Qwen3-235B-A22B model with SGLang on Ascend NPUs, including single-node, multi-node, and PD disaggregation modes." + description: "Deploy Qwen3-235B-A22B model with SGLang on Ascend NPUs, including single-node PD mixed and multi-node PD disaggregation modes." --- ## Introduction @@ -13,8 +13,8 @@ logical reasoning, text comprehension, mathematics, science, coding, and tool us and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-235B-A22B on Ascend NPUs using SGLang, including single-node PD mixed -mode, multi-node PD mixed mode, multi-node PD disaggregation mode, 256k long-sequence inference, Prefill Context Parallel, -feature configuration, and performance optimization. +mode, multi-node PD disaggregation mode, 256k long-sequence inference, Prefill Context Parallel, feature configuration, +and performance optimization. This document is validated and written based on **SGLang v0.5.13**. The current model (Qwen3-235B-A22B) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use @@ -29,8 +29,8 @@ v0.5.13 or a later version. | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend ascend_fuseep` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 94208` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` | +| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 94208` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -62,18 +62,25 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - Before downloading model weights, check the model size to reserve enough disk space. - +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Qwen3-235B-A22B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-235B-A22B-Instruct-2507) (BF16) -- [Qwen3-235B-A22B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-235B-A22B-W8A8) (W8A8 quantized version) +- [Qwen3-235B-A22B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-235B-A22B-Instruct-2507) (BF16, 470.21GB) +- [Qwen3-235B-A22B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-235B-A22B-W8A8) (W8A8 quantized version, 236.80GB) - [Qwen3-235B-A22B-Eagle3](https://www.modelscope.cn/models/nv-community/Qwen3-235B-A22B-Eagle3) (EAGLE3 draft model for speculative decoding) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +The BF16 variant (470.21GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full +A3 node (8 cards, 16 dies) or two A2 nodes. -It is recommended to download the model weights to a shared directory across multiple nodes. +The W8A8 variant (236.80GB) can be deployed on 8 × 64GB (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). + + +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation @@ -218,6 +225,7 @@ python3 -m sglang.launch_server \ --chunked-prefill-size -1 \ --skip-server-warmup \ --device npu \ + --quantization modelslim \ --tp-size 16 \ --mem-fraction-static 0.45 \ --max-running-requests 1 \ @@ -250,6 +258,7 @@ python3 -m sglang.launch_server \ --mem-fraction-static 0.8 \ --disable-cuda-graph \ --device npu \ + --quantization modelslim \ --disable-radix-cache \ --chunked-prefill-size 8192 \ --skip-server-warmup \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx index fd6e296df..b85f473da 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx @@ -26,7 +26,7 @@ v0.5.13 or a later version. | Tensor Parallelism | `--tp-size 2` | | Data Parallelism | `--dp-size 2` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -56,18 +56,22 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - Before downloading model weights, check the model size to reserve enough disk space. - +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Qwen3-30B-A3B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-30B-A3B-Instruct-2507) (BF16, recommended) -- [Qwen3-30B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8) (W8A8 quantized version) +- [Qwen3-30B-A3B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-30B-A3B-Instruct-2507) (BF16, 61.08GB, recommended) +- [Qwen3-30B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8) (W8A8 quantized version, 31.29GB) - [Qwen3-a3B_eagle3](https://www.modelscope.cn/models/vllm-ascend/Qwen3-a3B_eagle3) (EAGLE3 draft model for speculative decoding) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional cards. + +Both variants fit within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx index 808e918f4..14d9e0064 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx @@ -24,7 +24,7 @@ version. |-------------------------------|-----------------------------------------------------------------------------------------------| | Tensor Parallelism | `--tp-size 4` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 1 4 8 16` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 4 8 16` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -54,18 +54,24 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Qwen3-32B](https://www.modelscope.cn/models/Qwen/Qwen3-32B) (BF16) -- [Qwen3-32B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-32B-W8A8) (W8A8 quantized version) +- [Qwen3-32B](https://www.modelscope.cn/models/Qwen/Qwen3-32B) (BF16, 65.54GB) +- [Qwen3-32B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-32B-W8A8) (W8A8 quantized version, 42.77GB) - [Eagle3-Qwen3-32B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-32B-zh) (EAGLE3 draft model for speculative decoding) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional cards. + +The BF16 variant (65.54GB) can be deployed on 2 × 64GB of device memory (`--tp-size 2`), which corresponds to 2 cards on A2 or 1 card (2 dies) on A3. -It is recommended to download the model weights to a shared directory across multiple nodes. +The W8A8 variant (42.77GB) fits within a single die. + + +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx index ff7a213a5..7aa1596cb 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx @@ -28,7 +28,7 @@ use v0.5.13 or a later version. | Data Parallelism | `--dp-size 8` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -59,16 +59,21 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP) +- [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP, 235.88GB) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +The W4A8 variant (235.88GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 +node or 8 dies on A3 (4 cards). + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation @@ -186,6 +191,33 @@ curl http://${HOST}:${PORT}/generate \ Expected result: an HTTP 200 response with the generated text containing "Paris". +For multimodal requests (text + image): + +```shell +# ============================================================ +# Before running, update the following variables: +# HOST: the server host address (e.g., localhost) +# PORT: the server port number (e.g., 6688) +# ============================================================ + +curl http://${HOST}:${PORT}/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3.5-397B-A17B-w4a8-mtp", + "messages": [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, + {"type": "text", "text": "Describe this image."} + ] + } + ] + }' +``` + +Expected result: an HTTP 200 response with a description of the image. + Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx index 5600c29eb..f0ca7fad2 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx @@ -25,8 +25,8 @@ v0.5.13 or a later version. |-------------------------------|-----------------------------------------------------------------------------------------------| | Tensor Parallelism | `--tp-size 2` | | Quantization | `--quantization modelslim` | -| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 32768` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 8 16 32 48` | +| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 8 16 32 48` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -56,16 +56,20 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Eco-Tech/Qwen3.6-27B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) (W8A8 quantized version) +- [Eco-Tech/Qwen3.6-27B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) (W8A8 quantized version, 36.45GB) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional cards. + +The W8A8 variant (36.45GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation @@ -183,6 +187,33 @@ curl http://${HOST}:${PORT}/generate \ Expected result: an HTTP 200 response with the generated text containing "Paris". +For multimodal requests (text + image): + +```shell +# ============================================================ +# Before running, update the following variables: +# HOST: the server host address (e.g., localhost) +# PORT: the server port number (e.g., 6688) +# ============================================================ + +curl http://${HOST}:${PORT}/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3.6-27B-w8a8", + "messages": [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, + {"type": "text", "text": "Describe this image."} + ] + } + ] + }' +``` + +Expected result: an HTTP 200 response with a description of the image. + Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx index 3bf89235d..61810f5f1 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx @@ -25,8 +25,8 @@ v0.5.13 or a later version. | Feature | Example usage | |-------------------------------|-----------------------------------------------------------------------------------------------| | Tensor Parallelism | `--tp-size 2` | -| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g. `--chunked-prefill-size 16384` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` | +| Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -56,16 +56,20 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Eco-Tech/Qwen3.6-35B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) (W8A8 quantized version) +- [Eco-Tech/Qwen3.6-35B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) (W8A8 quantized version, 39.81GB) -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional cards. + +The W8A8 variant (39.81GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. + -It is recommended to download the model weights to a shared directory across multiple nodes. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. + +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation @@ -183,6 +187,33 @@ curl http://${HOST}:${PORT}/generate \ Expected result: an HTTP 200 response with the generated text containing "Paris". +For multimodal requests (text + image): + +```shell +# ============================================================ +# Before running, update the following variables: +# HOST: the server host address (e.g., localhost) +# PORT: the server port number (e.g., 6688) +# ============================================================ + +curl http://${HOST}:${PORT}/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen3.6-35B-A3B-w8a8", + "messages": [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, + {"type": "text", "text": "Describe this image."} + ] + } + ] + }' +``` + +Expected result: an HTTP 200 response with a description of the image. + Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx index 6a70a5c4b..d068e278d 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx @@ -27,7 +27,7 @@ v0.5.13 or a later version. | Data Parallelism | `--dp-size 2` | | Expert Parallelism | `--ep-size 4 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g. `--cuda-graph-bs 2 4 8` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 8` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant \`
`--speculative-draft-model-path /path/to/draft-model-weights` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -58,21 +58,23 @@ Before following this tutorial, complete the environment setup in the documents ### Model weights - -If you need to download model weights, check the model size before downloading to reserve enough space. - +Before downloading model weights, check the model size to reserve enough disk space. +For multi-node deployment, download the weights to a shared directory accessible to all nodes. -- [Qwen3-Next-80B-A3B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen3-Next-80B-A3B-Instruct) (BF16) -- [Qwen3-Next-80B-A3B-Instruct-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-Next-80B-A3B-Instruct-W8A8) (W8A8 quantized version) +- [Qwen3-Next-80B-A3B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen3-Next-80B-A3B-Instruct) (BF16, 162.68GB) — used as the EAGLE3 draft model +- [Qwen3-Next-80B-A3B-Instruct-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-Next-80B-A3B-Instruct-W8A8) (W8A8 quantized version, 84.90GB) - -For EAGLE3 speculative decoding, use the BF16 weights as the draft model (set `--speculative-draft-model-path` to the BF16 weight path). - + +We recommend deploying the W8A8 variant as the main model for reduced resource usage and higher throughput. +It (84.90GB) can be deployed on 2 × 64GB (`--tp-size 2`), which corresponds to 2 cards on A2 or 1 card (2 dies) on A3. +The BF16 weights serve as the EAGLE3 draft model (set `--speculative-draft-model-path` to the BF16 weight path). + -Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, -refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. +This is the minimum recommended configuration. For optimized configurations, +see [Best practices](#best-practices), which may require additional cards or nodes. -It is recommended to download the model weights to a shared directory across multiple nodes. +For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation
✅
Alibaba-NLP/gme-Qwen2-VL-2B-Instructiic/gme-Qwen2-VL-2B-Instruct GME (Multimodal) ✅ ✅