📝 docs(diffusion): add MXFP4 quantization docs (#25904)

This commit is contained in:
Junlin Wu
2026-05-25 10:24:30 +03:00
committed by GitHub
parent ca029e816b
commit aae04b1241
2 changed files with 54 additions and 1 deletions
@@ -52,6 +52,14 @@ SGLang support **mix-bits** quantization (independently defines and loads each l
<td><strong style={{color: 'green'}}>√</strong></td> <td><strong style={{color: 'green'}}>√</strong></td>
<td><strong style={{color: 'green'}}>√</strong></td> <td><strong style={{color: 'green'}}>√</strong></td>
</tr> </tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/22338">MXFP4</a></td>
<td>Linear</td>
<td><strong style={{color: 'red'}}>x</strong></td>
<td><strong style={{color: 'red'}}>x</strong></td>
<td><strong style={{color: 'green'}}>√</strong></td>
<td><strong style={{color: 'green'}}>√</strong></td>
</tr>
<tr> <tr>
<td>W4A4 dynamic</td> <td>W4A4 dynamic</td>
<td>MoE</td> <td>MoE</td>
@@ -102,7 +102,7 @@ backend.
<td><code>--model-path</code></td> <td><code>--model-path</code></td>
<td>Wan2.2 family</td> <td>Wan2.2 family</td>
<td>None</td> <td>None</td>
<td>Currently only compatible with the Ascend NPU family and supports <code>mxfp8</code>, <code>w8a8</code>, and <code>w4a4</code></td> <td>Currently only compatible with the Ascend NPU family and supports <code>mxfp8</code>, <code>mxfp4</code>, <code>w8a8</code>, and <code>w4a4</code></td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@@ -601,6 +601,8 @@ MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression t
- [x] ```W8A8_DYNAMIC``` linear with online quantization of activations - [x] ```W8A8_DYNAMIC``` linear with online quantization of activations
- [x] ```W8A8_MXFP8``` linear with offline quantization (msmodelslim pre-quantized weights) - [x] ```W8A8_MXFP8``` linear with offline quantization (msmodelslim pre-quantized weights)
- [x] ```mxfp8``` linear with online quantization (`--quantization mxfp8`) - [x] ```mxfp8``` linear with online quantization (`--quantization mxfp8`)
- [x] ```W4A4_MXFP4``` / ```W4A4_MXFP4_DUALSCALE``` linear with offline quantization (msmodelslim pre-quantized weights)
- [x] ```mxfp4_npu``` linear with online quantization (`--quantization mxfp4_npu`)
## MXFP8 Online Quantization ## MXFP8 Online Quantization
@@ -630,3 +632,46 @@ sglang generate \
--prompt "a beautiful sunset" \ --prompt "a beautiful sunset" \
--save-output --save-output
``` ```
## MXFP4 Online Quantization
For online MXFP4 quantization on Ascend NPU, load the original FP16/BF16 model and add
`--quantization mxfp4_npu`. The `mxfp4_npu` key is used for Ascend because `mxfp4`
is reserved for the ROCm/aiter backend.
Weights are quantized at load time via `npu_dynamic_dual_level_mx_quant`, and activations
are quantized per-token during inference before `npu_dual_level_quant_matmul`. MXFP4 uses
dual-level block scales with an L1 block size of 32 and an L0 block size of 512.
```bash
sglang generate \
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--quantization mxfp4_npu \
--prompt "a fox walking through neon rain" \
--save-output
```
> **Hardware requirement:** Ascend A5 series or newer. `npu_dynamic_dual_level_mx_quant`
> and `npu_dual_level_quant_matmul` are not available on A2/A3.
>
> **Note:** Online MXFP4 weight quantization is experimental. The offline msmodelslim
> flow uses pre-quantized weights and may produce different numerical results.
## MXFP4 Offline Quantization (msmodelslim)
Pre-quantized MXFP4 weights exported by msmodelslim are auto-detected via
`quant_model_description.json` (`W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` scheme).
Use `wan_repack.py` to convert the quantized weights to Diffusers format, then load
the converted model with `--model-path`:
```bash
sglang generate \
--model-path {path_to_converted_mxfp4_model} \
--prompt "a beautiful sunset" \
--save-output
```
The offline MXFP4 checkpoint stores weights in an FP8 container and includes dual-level
scales (`weight_scale`, `weight_dual_scale`). If exported with smooth quantization,
`mul_scale` is loaded and applied before activation quantization to keep activations
aligned with the calibrated weights.