minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU] Support GGUF quantization for Ascend NPU (dense + MoE) (#17883)

Browse Source 
Co-authored-by: ronnie_zheng <zl19940307@163.com>
This commit is contained in:
1874.
2026-04-25 17:16:47 +03:00
committed by GitHub
co-authored by ronnie_zheng
parent e708ea6d94
commit 046c14a3ed
12 changed files with 834 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/platforms/ascend/ascend_npu_quantization.md
+5 -1
View File
@@ -48,5 +48,9 @@ Compressed-tensors (LLM Compressor) on Ascend support:
| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|-----------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
| [GGUF (all types)](https://github.com/sgl-project/sglang/pull/17883) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
| [GGUF (all types)](https://github.com/sgl-project/sglang/pull/17883) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
in progress
> Note: On Ascend, GGUF weights are pre-dequantized to FP16/BF16 during model loading to ensure optimal inference performance. This enables support for all GGUF quantization types (Q2_K, Q4_K_M, IQ4_XS, etc.) while maintaining high inference speed.