[diffusion] refactor: reuse srt quantization contracts and mxfp8 kernels (#36063)

This commit is contained in:
Mick
2026-08-24 09:26:54 +08:00
committed by GitHub
parent 167c339c8e
commit 1c1c9d9b4e
16 changed files with 277 additions and 511 deletions
@@ -291,6 +291,13 @@ the fused ConvRot kernel without requantizing them. TP1/2/4 and sequence or
layerwise offload are supported; TP8 violates the checkpoint's 256-element
ConvRot group boundary, and FSDP is rejected.
Self-describing community MXFP8 files use the same
`--transformer-weights-path <repo/file.safetensors>` form and need no
quantization flag. SGLang reads their global format metadata, keeps unmarked H3
layers in their original dtype, and reuses SRT's MXFP8 dense kernels. The
selected SRT backend must support MXFP8 on the target GPU; FSDP is rejected for
this mixed per-layer layout.
### Advanced: precomputed AdaLN cache
The [model card](https://huggingface.co/MiniMaxAI/MiniMax-H3) notes that about
+17 -5
View File
@@ -176,6 +176,14 @@ backend.
<td><code>comfy-kitchen</code></td>
<td>CUDA; auto-detected; uses the fused Kitchen INT8 kernel and validates weight/scale layout before model construction. TP requires every row-parallel input shard to preserve the checkpoint's ConvRot group boundary. The H3 256-group DiT supports TP1/2/4, not TP8; its Qwen3-VL encoder keeps TP8 by replicating only incompatible row projections. Offload is supported; FSDP is not.</td>
</tr>
<tr>
<td><code>mxfp8</code></td>
<td>Self-describing serialized weights, or BF16/FP16 weights for online quantization</td>
<td><code>--transformer-weights-path</code>, or <code>--quantization mxfp8</code> for online quantization</td>
<td>Native diffusion linears; mixed MiniMax-H3 checkpoints are selected per layer</td>
<td>SRT's platform MXFP8 backend</td>
<td>Serialized metadata is auto-detected. NVIDIA and ROCm reuse SRT's dense MXFP8 kernels; Ascend keeps its native online path. Mixed per-layer checkpoints do not support FSDP.</td>
</tr>
<tr>
<td><code>qvg-kv</code></td>
<td>Unquantized model with runtime causal KV-cache compression</td>
@@ -1124,11 +1132,13 @@ MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression t
- [x] ```W4A4_MXFP4``` / ```W4A4_MXFP4_DUALSCALE``` linear with offline quantization (msmodelslim pre-quantized weights)
- [x] ```mxfp4_npu``` linear with online quantization (`--quantization mxfp4_npu`)
## MXFP8 Online Quantization
## MXFP8 Quantization
For online MXFP8 quantization, load the original FP16/BF16 model and add `--quantization mxfp8`.
Weights are quantized at load time via `npu_dynamic_mx_quant`, and activations are quantized per-token
during inference with `npu_quant_matmul` (block_size=32).
Self-describing serialized MXFP8 transformers are inferred from their
checkpoint metadata. For online MXFP8 quantization, load the original
FP16/BF16 model and add `--quantization mxfp8`. NVIDIA and ROCm reuse SRT's
dense MXFP8 implementation; Ascend uses `npu_dynamic_mx_quant` and
`npu_quant_matmul` with a block size of 32.
```bash
sglang generate \
@@ -1138,7 +1148,9 @@ sglang generate \
--save-output
```
> **Hardware requirement:** Ascend A5 series or newer. `npu_dynamic_mx_quant` is not available on A2/A3.
The selected SRT backend must provide an MXFP8 kernel on NVIDIA or ROCm. On
Ascend, the hardware requirement remains A5 series or newer;
`npu_dynamic_mx_quant` is not available on A2/A3.
## MXFP8 Offline Quantization (msmodelslim)