From 47b8eadbc4dd7fe8440f7effb2749389802e3995 Mon Sep 17 00:00:00 2001 From: "1874." <1094918@gmail.com> Date: Mon, 27 Apr 2026 22:30:24 +0800 Subject: [PATCH] [Docs] Update Ascend NPU GGUF quantization documentation (#23845) --- .../docs/advanced_features/quantization.mdx | 4 +- .../ascend-npus/ascend_npu_quantization.mdx | 208 +++++++++++------- .../ascend_npu_support_features.mdx | 2 +- 3 files changed, 133 insertions(+), 81 deletions(-) diff --git a/docs_new/docs/advanced_features/quantization.mdx b/docs_new/docs/advanced_features/quantization.mdx index e13ca084e..bd59a0497 100644 --- a/docs_new/docs/advanced_features/quantization.mdx +++ b/docs_new/docs/advanced_features/quantization.mdx @@ -130,8 +130,8 @@ The following table summarizes quantization method support across NVIDIA and AMD gguf Yes No - WIP - CUDA-only kernels in sgl-kernel + Yes + CUDA kernels in sgl-kernel; Ascend uses CPU pre-dequantization at load time modelopt / modelopt_fp8 diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx index 5d3c0b77b..c952a3606 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx @@ -23,66 +23,66 @@ SGLang support **mix-bits** quantization (independently defines and loads each l W4A4 dynamic Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> - <span style="color: green;">√</span> + √ + √ + TBD + √ W8A8 static Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> - <span style="color: green;">√</span> + √ + √ + TBD + √ W8A8 dynamic Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> - <span style="color: green;">√</span> + √ + √ + TBD + √ MXFP8 Linear - <span style="color: red;">x</span> - <span style="color: red;">x</span> - <span style="color: blue;">WIP</span> - <span style="color: blue;">WIP</span> + x + x + WIP + WIP W4A4 dynamic MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> - <span style="color: red;">x</span> + √ + √ + TBD + x W4A8 dynamic MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> - <span style="color: red;">x</span> + √ + √ + TBD + x W8A8 dynamic MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> - <span style="color: red;">x</span> + √ + √ + TBD + x MXFP8 MoE - <span style="color: red;">x</span> - <span style="color: red;">x</span> - <span style="color: blue;">WIP</span> - <span style="color: red;">x</span> + x + x + WIP + x @@ -102,23 +102,23 @@ SGLang support **mix-bits** quantization (independently defines and loads each l W4A16 Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W8A16 Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W4A16 MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD @@ -138,30 +138,30 @@ GPTQ on Ascend support W4A16 Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W8A16 Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W4A16 MOE MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W8A16 MOE MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD @@ -181,30 +181,30 @@ GPTQ on Ascend support W4A16 Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W8A16 Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W4A16 MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W8A16 MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD @@ -224,34 +224,86 @@ Compressed-tensors (LLM Compressor) on Ascend support: W8A8 dynamic Linear - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W4A8 dynamic with/without activation clip MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W4A16 MOE MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD W8A8 dynamic MoE - <span style="color: green;">√</span> - <span style="color: green;">√</span> - <span style="color: yellow;">TBD</span> + √ + √ + TBD [GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883) + + + + + + + + + + + + + + + + + + + + + + + + + + +
Quantization typeLayer typeA2 SupportedA3 SupportedA5 Supported
All GGUF types (standard, K-quant)Linear√√TBD
All GGUF types (standard, K-quant)MoE√√TBD
-in progress +**Usage Examples:** + +- Dense model (e.g. Qwen3-14B-Q4_K_M.gguf): + +```bash Command +python3 -m sglang.launch_server \ + --model-path Qwen3-14B-Q4_K_M.gguf \ + --device npu --attention-backend ascend \ + --host 0.0.0.0 --port 30000 \ + --mem-fraction-static 0.7 --tp-size 2 +``` + +- MoE model (e.g. Qwen3-30B-A3B-Q4_K_M.gguf): + +```bash Command +python3 -m sglang.launch_server \ + --model-path Qwen3-30B-A3B-Q4_K_M.gguf \ + --device npu --attention-backend ascend \ + --host 0.0.0.0 --port 30000 \ + --mem-fraction-static 0.8 --tp-size 2 +``` + +> **Implementation Notes:** +> - GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead). +> - MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation. +> - TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx index dfcbc451b..884d8f1dc 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx @@ -61,7 +61,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--load-format` `auto` - auto, safetensors + auto, safetensors, gguf A2, A3