[diffusion] refactor: reuse srt quantization contracts and mxfp8 kernels (#36063)
This commit is contained in:
@@ -291,6 +291,13 @@ the fused ConvRot kernel without requantizing them. TP1/2/4 and sequence or
|
||||
layerwise offload are supported; TP8 violates the checkpoint's 256-element
|
||||
ConvRot group boundary, and FSDP is rejected.
|
||||
|
||||
Self-describing community MXFP8 files use the same
|
||||
`--transformer-weights-path <repo/file.safetensors>` form and need no
|
||||
quantization flag. SGLang reads their global format metadata, keeps unmarked H3
|
||||
layers in their original dtype, and reuses SRT's MXFP8 dense kernels. The
|
||||
selected SRT backend must support MXFP8 on the target GPU; FSDP is rejected for
|
||||
this mixed per-layer layout.
|
||||
|
||||
### Advanced: precomputed AdaLN cache
|
||||
|
||||
The [model card](https://huggingface.co/MiniMaxAI/MiniMax-H3) notes that about
|
||||
|
||||
@@ -176,6 +176,14 @@ backend.
|
||||
<td><code>comfy-kitchen</code></td>
|
||||
<td>CUDA; auto-detected; uses the fused Kitchen INT8 kernel and validates weight/scale layout before model construction. TP requires every row-parallel input shard to preserve the checkpoint's ConvRot group boundary. The H3 256-group DiT supports TP1/2/4, not TP8; its Qwen3-VL encoder keeps TP8 by replicating only incompatible row projections. Offload is supported; FSDP is not.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>mxfp8</code></td>
|
||||
<td>Self-describing serialized weights, or BF16/FP16 weights for online quantization</td>
|
||||
<td><code>--transformer-weights-path</code>, or <code>--quantization mxfp8</code> for online quantization</td>
|
||||
<td>Native diffusion linears; mixed MiniMax-H3 checkpoints are selected per layer</td>
|
||||
<td>SRT's platform MXFP8 backend</td>
|
||||
<td>Serialized metadata is auto-detected. NVIDIA and ROCm reuse SRT's dense MXFP8 kernels; Ascend keeps its native online path. Mixed per-layer checkpoints do not support FSDP.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>qvg-kv</code></td>
|
||||
<td>Unquantized model with runtime causal KV-cache compression</td>
|
||||
@@ -1124,11 +1132,13 @@ MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression t
|
||||
- [x] ```W4A4_MXFP4``` / ```W4A4_MXFP4_DUALSCALE``` linear with offline quantization (msmodelslim pre-quantized weights)
|
||||
- [x] ```mxfp4_npu``` linear with online quantization (`--quantization mxfp4_npu`)
|
||||
|
||||
## MXFP8 Online Quantization
|
||||
## MXFP8 Quantization
|
||||
|
||||
For online MXFP8 quantization, load the original FP16/BF16 model and add `--quantization mxfp8`.
|
||||
Weights are quantized at load time via `npu_dynamic_mx_quant`, and activations are quantized per-token
|
||||
during inference with `npu_quant_matmul` (block_size=32).
|
||||
Self-describing serialized MXFP8 transformers are inferred from their
|
||||
checkpoint metadata. For online MXFP8 quantization, load the original
|
||||
FP16/BF16 model and add `--quantization mxfp8`. NVIDIA and ROCm reuse SRT's
|
||||
dense MXFP8 implementation; Ascend uses `npu_dynamic_mx_quant` and
|
||||
`npu_quant_matmul` with a block size of 32.
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
@@ -1138,7 +1148,9 @@ sglang generate \
|
||||
--save-output
|
||||
```
|
||||
|
||||
> **Hardware requirement:** Ascend A5 series or newer. `npu_dynamic_mx_quant` is not available on A2/A3.
|
||||
The selected SRT backend must provide an MXFP8 kernel on NVIDIA or ROCm. On
|
||||
Ascend, the hardware requirement remains A5 series or newer;
|
||||
`npu_dynamic_mx_quant` is not available on A2/A3.
|
||||
|
||||
## MXFP8 Offline Quantization (msmodelslim)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user