📝 docs(diffusion): add MXFP4 quantization docs (#25904)
This commit is contained in:
@@ -52,6 +52,14 @@ SGLang support **mix-bits** quantization (independently defines and loads each l
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><a href="https://github.com/sgl-project/sglang/pull/22338">MXFP4</a></td>
|
||||
<td>Linear</td>
|
||||
<td><strong style={{color: 'red'}}>x</strong></td>
|
||||
<td><strong style={{color: 'red'}}>x</strong></td>
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>W4A4 dynamic</td>
|
||||
<td>MoE</td>
|
||||
|
||||
@@ -102,7 +102,7 @@ backend.
|
||||
<td><code>--model-path</code></td>
|
||||
<td>Wan2.2 family</td>
|
||||
<td>None</td>
|
||||
<td>Currently only compatible with the Ascend NPU family and supports <code>mxfp8</code>, <code>w8a8</code>, and <code>w4a4</code></td>
|
||||
<td>Currently only compatible with the Ascend NPU family and supports <code>mxfp8</code>, <code>mxfp4</code>, <code>w8a8</code>, and <code>w4a4</code></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -601,6 +601,8 @@ MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression t
|
||||
- [x] ```W8A8_DYNAMIC``` linear with online quantization of activations
|
||||
- [x] ```W8A8_MXFP8``` linear with offline quantization (msmodelslim pre-quantized weights)
|
||||
- [x] ```mxfp8``` linear with online quantization (`--quantization mxfp8`)
|
||||
- [x] ```W4A4_MXFP4``` / ```W4A4_MXFP4_DUALSCALE``` linear with offline quantization (msmodelslim pre-quantized weights)
|
||||
- [x] ```mxfp4_npu``` linear with online quantization (`--quantization mxfp4_npu`)
|
||||
|
||||
## MXFP8 Online Quantization
|
||||
|
||||
@@ -630,3 +632,46 @@ sglang generate \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
## MXFP4 Online Quantization
|
||||
|
||||
For online MXFP4 quantization on Ascend NPU, load the original FP16/BF16 model and add
|
||||
`--quantization mxfp4_npu`. The `mxfp4_npu` key is used for Ascend because `mxfp4`
|
||||
is reserved for the ROCm/aiter backend.
|
||||
|
||||
Weights are quantized at load time via `npu_dynamic_dual_level_mx_quant`, and activations
|
||||
are quantized per-token during inference before `npu_dual_level_quant_matmul`. MXFP4 uses
|
||||
dual-level block scales with an L1 block size of 32 and an L0 block size of 512.
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--quantization mxfp4_npu \
|
||||
--prompt "a fox walking through neon rain" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
> **Hardware requirement:** Ascend A5 series or newer. `npu_dynamic_dual_level_mx_quant`
|
||||
> and `npu_dual_level_quant_matmul` are not available on A2/A3.
|
||||
>
|
||||
> **Note:** Online MXFP4 weight quantization is experimental. The offline msmodelslim
|
||||
> flow uses pre-quantized weights and may produce different numerical results.
|
||||
|
||||
## MXFP4 Offline Quantization (msmodelslim)
|
||||
|
||||
Pre-quantized MXFP4 weights exported by msmodelslim are auto-detected via
|
||||
`quant_model_description.json` (`W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` scheme).
|
||||
Use `wan_repack.py` to convert the quantized weights to Diffusers format, then load
|
||||
the converted model with `--model-path`:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path {path_to_converted_mxfp4_model} \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
The offline MXFP4 checkpoint stores weights in an FP8 container and includes dual-level
|
||||
scales (`weight_scale`, `weight_dual_scale`). If exported with smooth quantization,
|
||||
`mul_scale` is loaded and applied before activation quantization to keep activations
|
||||
aligned with the calibrated weights.
|
||||
|
||||
Reference in New Issue
Block a user