[AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs (#29328)

This commit is contained in:
Colin Z
2026-08-14 21:59:39 -07:00
committed by GitHub
parent 5afdb1caea
commit bc7e3ba66c
14 changed files with 1218 additions and 204 deletions
+19 -2
View File
@@ -865,7 +865,7 @@ Other layers (e.g. projections in the attention layers) have their weights quant
### `quark_mxfp4` online quantization method
SGLang running on AMD GPUs with hardware FP4 support (CDNA4 architecture, e.g. MI355x) supports the quantization method `--quantization quark_mxfp4`, that will quantize BF16 model weights to MXFP4 at load time, use dynamic MXFP4 quantization for activations and MXFP4 GEMMs instead of BF16 GEMMs.
SGLang running on AMD GPUs with hardware FP4 support (CDNA4 architecture, e.g. MI355x) supports the quantization method `--quantization quark_mxfp4`, that will quantize BF16 or NVFP4 model weights to MXFP4 at load time, use dynamic MXFP4 quantization for activations and MXFP4 GEMMs instead of BF16 GEMMs.
Example (BF16 to MXFP4 requantization):
@@ -875,7 +875,24 @@ sglang serve --model-path Qwen/Qwen3-30B-A3B \
--quantization quark_mxfp4
```
The option `--quantization quark_mxfp4` also supports converting FP8 dense and MOE models to MXFP4, following this logic:
#### Online NVFP4 to MXFP4 Requantization
The option `--quantization quark_mxfp4` supports converting NVFP4 checkpoints (e.g. `nvidia/Kimi-K2.6-NVFP4`) to MXFP4 at load time to allow efficient inference using supported AMD hardware (gfx95x+):
- The quantization metadata of the source NVFP4 checkpoint is read from either `config.json` (`quantization_config`) or a standalone `hf_quant_config.json`;
- Producer-declared excluded modules will remain in higher precision;
- NVFP4 checkpoints with mixed precision (`"quant_algo": "MIXED_PRECISION"`, e.g. `nvidia/Qwen3.5-397B-A17B-NVFP4-V2`) are also supported.
Example (NVFP4 to MXFP4 requantization):
```bash
sglang serve --model-path nvidia/Kimi-K2.6-NVFP4 \
--tensor-parallel-size 4 \
--quantization quark_mxfp4 \
```
#### Online FP8 to MXFP4 Requantization
The option `--quantization quark_mxfp4` supports converting FP8 dense and MOE models to MXFP4, following this logic:
1. Load an FP8 weight tensor,
2. Dequantize it to BF16,