minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[AMD][MXFP4] Reland "Online MXFP4 quantization 2/N - FP8 to MXFP4 requantization on AMD GPUs" (#28291)

Browse Source 
Co-authored-by: Bowen Bao <bowenbao@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
fxmarty-amd
2026-07-21 02:50:23 -07:00
committed by GitHub
co-authored by Bowen Bao HAI Claude
parent 696e8f80d1
commit dcd9014f15
14 changed files with 1052 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/advanced_features/quantization.mdx
+17 -1
View File
@@ -883,7 +883,7 @@ Other layers (e.g. projections in the attention layers) have their weights quant
SGLang running on AMD GPUs with hardware FP4 support (CDNA4 architecture, e.g. MI355x) supports the quantization method `--quantization quark_mxfp4`, that will quantize BF16 model weights to MXFP4 at load time, use dynamic MXFP4 quantization for activations and MXFP4 GEMMs instead of BF16 GEMMs.
Example:
Example (BF16 to MXFP4 requantization):
```bash
sglang serve --model-path Qwen/Qwen3-30B-A3B \
@@ -891,6 +891,22 @@ sglang serve --model-path Qwen/Qwen3-30B-A3B \
--quantization quark_mxfp4
```
The option `--quantization quark_mxfp4` also supports converting FP8 dense and MOE models to MXFP4, following this logic:
1. Load an FP8 weight tensor,
2. Dequantize it to BF16,
3. Requantize it to MXFP4
progressively during weight loading.
Example (FP8 to MXFP4 requantization):
```bash
sglang serve --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
--tensor-parallel-size 1 \
--quantization quark_mxfp4
```
### Intel® Neural Compressor online quantization method
SGLang supports quantization methods based on the advanced algorithm [auto-round](https://github.com/intel/auto-round) in [Intel® Neural Compressor](https://github.com/intel/neural-compressor). You can simply specify `--quantization auto-round-int8` to use this feature. It will quantize the model on the fly to target format. More online quantization methods are on the way.