minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU] Support GGUF quantization for Ascend NPU (dense + MoE) (#17883)

Browse Source 
Co-authored-by: ronnie_zheng <zl19940307@163.com>
This commit is contained in:
1874.
2026-04-25 17:16:47 +03:00
committed by GitHub
co-authored by ronnie_zheng
parent e708ea6d94
commit 046c14a3ed
12 changed files with 834 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/quantization.md
+1 -1
View File
@@ -36,7 +36,7 @@ The following table summarizes quantization method support across NVIDIA and AMD
| `quark_int4fp8_moe` | No | Yes | No | AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4) |
| `awq_marlin` | Yes | No | No | Marlin kernels are CUDA-only |
| `gptq_marlin` | Yes | No | No | Marlin kernels are CUDA-only |
| `gguf` | Yes | No | WIP | CUDA-only kernels in sgl-kernel |
| `gguf` | Yes | No | Yes | CUDA-only kernels in sgl-kernel; Pre-dequantized on Ascend |
| `modelopt` / `modelopt_fp8` | Yes (Hopper/SM90+) | No | No | [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer); requires NVIDIA hardware |
| `modelopt_fp4` | Yes (Blackwell/SM100+) | No | No | [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer); native FP4 on Blackwell (B200, GB200) |
| `petit_nvfp4` | No | Yes (MI250/MI300X/MI325X) | No | Enables NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel); use `modelopt_fp4` on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/) and [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html). |
docs/platforms/ascend/ascend_npu_quantization.md
+5 -1
View File
@@ -48,5 +48,9 @@ Compressed-tensors (LLM Compressor) on Ascend support:
| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|-----------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
| [GGUF (all types)](https://github.com/sgl-project/sglang/pull/17883) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
| [GGUF (all types)](https://github.com/sgl-project/sglang/pull/17883) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
in progress
> Note: On Ascend, GGUF weights are pre-dequantized to FP16/BF16 during model loading to ensure optimal inference performance. This enables support for all GGUF quantization types (Q2_K, Q4_K_M, IQ4_XS, etc.) while maintaining high inference speed.