[diffusion] quant: support gguf (#35370)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Mick <mickjagger19@icloud.com>
This commit is contained in:
zijiexia
co-authored by
Claude Fable 5
Mick
Claude Fable 5
Mick
parent
ae23423b46
commit
21c88f8625
@@ -198,6 +198,50 @@ baseline; everything else stays identical. GPU peak stays about 18 GB
|
||||
either way because streaming offload is set by the offload buffers and VAE
|
||||
decode, not the weight dtype.
|
||||
|
||||
### Pre-quantized GGUF transformer
|
||||
|
||||
Use `--transformer-weights-path` to replace only the DiT with a GGUF file; the
|
||||
base repository continues to provide the text encoder, VAEs, scheduler, and
|
||||
tokenizers. Do not also pass `--quantization gguf`.
|
||||
|
||||
```bash 1×RTX 5090 Q4_K_M
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-H3 \
|
||||
--model-variant fl2va \
|
||||
--transformer-weights-path \
|
||||
leejet/MiniMax-H3-GGUF/minimax_h3_fl2va-Q4_K_M.gguf \
|
||||
--attention-backend fa \
|
||||
--performance-mode memory \
|
||||
--layerwise-offload-components dit,text_encoder \
|
||||
--dit-offload-prefetch-size 1 \
|
||||
--dit-layerwise-resident-layers 0 \
|
||||
--enable-torch-compile false \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
The loader also recognizes pruned checkpoints that replace the timestep MLP
|
||||
with `adaln_t_table`. Repositories containing both FL2VA and Ref2VA variants
|
||||
need a full file reference:
|
||||
|
||||
```bash Pruned FL2VA Q4_K
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-H3 \
|
||||
--model-variant fl2va \
|
||||
--transformer-weights-path \
|
||||
unsloth/MiniMax-H3-GGUF/minimax_h3_fl2va_pruned-Q4_K.gguf \
|
||||
--performance-mode memory \
|
||||
--layerwise-offload-components dit,text_encoder \
|
||||
--enable-torch-compile false \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
The linear adapter reuses SRT's GGUF type definitions and CUDA dequantization,
|
||||
then runs the native GEMM. SRT's fused MMVQ/MMQ kernels target the low-token LLM
|
||||
regime and are slower at diffusion sequence lengths. TP is supported when each
|
||||
row-parallel input shard remains GGML-block aligned; incompatible degrees fail
|
||||
during model construction. FSDP, LoRA merging, and the separate MiniMax-H3
|
||||
AdaLN cache flags are not compatible with packed GGUF weights.
|
||||
|
||||
The first launch downloads the model through the selected Hub. If the Hugging
|
||||
Face repository requires authentication, export a Hugging Face token in the
|
||||
server environment.
|
||||
|
||||
@@ -125,7 +125,8 @@ For quantized transformer checkpoints, prefer:
|
||||
|
||||
- `--model-path` for the base pipeline
|
||||
- `--transformer-path` for a quantized `transformers` transformer component folder
|
||||
- `--transformer-weights-path` for a quantized safetensors file, directory, or repo
|
||||
- `--transformer-weights-path` for a quantized safetensors file, directory,
|
||||
repo, or a supported GGUF transformer file
|
||||
- `--quantization` for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically)
|
||||
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
|
||||
|
||||
|
||||
@@ -135,7 +135,7 @@ Rows are grouped when a family shares the same runtime path or optimization supp
|
||||
<td>MiniMax-H3</td>
|
||||
<td><div className="sgd-id-list"><code>MiniMaxAI/MiniMax-H3</code></div></td>
|
||||
<td>T2VA / FL2VA / Ref2VA, 768p at 24 fps with synchronized audio</td>
|
||||
<td><span className="sgd-chip">Cache-DiT</span><span className="sgd-chip">Sage</span><span className="sgd-chip">Online FP8</span></td>
|
||||
<td><span className="sgd-chip">Cache-DiT</span><span className="sgd-chip">Sage</span><span className="sgd-chip">Online FP8</span><span className="sgd-chip">GGUF</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Wan2.1 Fun</td>
|
||||
|
||||
@@ -135,6 +135,14 @@ backend.
|
||||
<td>None</td>
|
||||
<td>Currently only compatible with the Ascend NPU family and supports <code>mxfp8</code>, <code>mxfp4</code>, <code>w8a8</code>, and <code>w4a4</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>gguf</code></td>
|
||||
<td>A single community <code>.gguf</code> holding the transformer</td>
|
||||
<td><code>--transformer-weights-path</code></td>
|
||||
<td>MiniMax-H3 <code>fl2va</code> (original and pruned AdaLN curve)</td>
|
||||
<td>None</td>
|
||||
<td>CUDA only, no FSDP. TP requires GGML-aligned shard boundaries. Shrinks the download and the host memory offload pins (17.5 vs 61.7 GiB for H3) rather than peak VRAM, which offload already bounds. Dequantized per use, so it is not faster. See <a href="#gguf">GGUF</a>.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -672,6 +680,131 @@ sglang generate \
|
||||
`quant_algo=NVFP4`; the `modelopt-nvfp4` label here is again a documentation
|
||||
family name rather than a serialized config key.
|
||||
|
||||
## GGUF
|
||||
|
||||
GGUF loads a community-quantized transformer from a single `.gguf` file while
|
||||
the rest of the pipeline — VAE, text encoder, scheduler, tokenizer — keeps
|
||||
loading from the base model.
|
||||
|
||||
GGUF primarily reduces checkpoint, host-memory, and resident-weight size. For
|
||||
example, MiniMax-H3's transformer is 17.5 GiB as Q4_K_M versus 61.7 GiB as
|
||||
BF16. With full layerwise offload, VAE decode and offload buffers can still
|
||||
dominate peak GPU memory, but each streamed DiT layer also transfers fewer
|
||||
bytes.
|
||||
|
||||
Packed linears reuse SRT's GGUF type definitions and CUDA dequantization, then
|
||||
run the native GEMM. SRT's fused MMVQ/MMQ kernels target the low-token LLM
|
||||
regime and are slower at diffusion sequence lengths. GGUF remains a
|
||||
capacity-oriented option; latency depends on the quantization type, activation
|
||||
shape, and placement policy.
|
||||
|
||||
Layers the checkpoint stores unquantized (F32/F16/BF16) take the ordinary linear
|
||||
path rather than the packed one. Their precision is then whatever the model
|
||||
declares for that layer, exactly as on the safetensors path — a checkpoint
|
||||
cannot raise a layer above the model's own dtype by storing it wider. For
|
||||
MiniMax-H3 the two agree: the layers it pins to FP32 are the ones the validated
|
||||
checkpoint leaves unquantized, and `post_load_weights` fails the load if any of
|
||||
them ends up narrower.
|
||||
|
||||
### Usage
|
||||
|
||||
No extra install: `gguf` is already a core SGLang dependency.
|
||||
|
||||
`--model-path` stays the base model; `--transformer-weights-path` takes the
|
||||
GGUF. A local path, `owner/repo/file.gguf`, or `owner/repo:QUANT_TYPE` all work.
|
||||
The quant-type shorthand is accepted only when exactly one repository file
|
||||
matches it; otherwise SGLang lists the candidates and asks for a full path.
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-H3 \
|
||||
--model-variant fl2va \
|
||||
--transformer-weights-path \
|
||||
leejet/MiniMax-H3-GGUF/minimax_h3_fl2va-Q4_K_M.gguf \
|
||||
--num-gpus 1 \
|
||||
--attention-backend fa \
|
||||
--performance-mode memory \
|
||||
--layerwise-offload-components dit,text_encoder \
|
||||
--dit-offload-prefetch-size 1 \
|
||||
--dit-layerwise-resident-layers 0 \
|
||||
--enable-torch-compile false \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
Note that `--quantization gguf` is not the selector — the quantization is read
|
||||
from the file itself, so passing the file is what enables the path.
|
||||
|
||||
MiniMax-H3 GGUF checkpoints may use either the original timestep MLP or the
|
||||
pruned AdaLN curve architecture. Repositories often contain both FL2VA and
|
||||
Ref2VA files, so use the full Hub file reference instead of an ambiguous
|
||||
`owner/repo:QUANT_TYPE` selector:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-H3 \
|
||||
--model-variant fl2va \
|
||||
--transformer-weights-path \
|
||||
unsloth/MiniMax-H3-GGUF/minimax_h3_fl2va_pruned-Q4_K.gguf \
|
||||
--performance-mode memory \
|
||||
--layerwise-offload-components dit,text_encoder \
|
||||
--enable-torch-compile false \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
The pruned architecture keeps its sampled curve and reduced AdaLN projections
|
||||
in FP32, matching the published checkpoint implementation. This precision
|
||||
island deliberately bypasses the BF16-only fused modulation kernels.
|
||||
|
||||
#### Whether to offload the VAE
|
||||
|
||||
Adding `vae` to `--layerwise-offload-components` trades a lot of latency for
|
||||
some peak VRAM, because the video VAE decoder is re-streamed per decode tile.
|
||||
Measured on 1× RTX 5090 with this checkpoint, 1344×768 × 107 frames:
|
||||
|
||||
| `--layerwise-offload-components` | Peak VRAM | Denoise | VAE decode |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| `dit,text_encoder` | 26.3 GiB | 39.0 s | **9.5 s** |
|
||||
| `dit,text_encoder,vae` | **19.6 GiB** | 39.0 s | 57.2 s |
|
||||
|
||||
Denoise is unaffected, and the output is bit-identical either way. Leave the VAE
|
||||
resident unless the 6.7 GiB matters — on a 24 GB card by this measurement it
|
||||
does, and the 6× slower decode is the price of fitting.
|
||||
|
||||
### Constraints
|
||||
|
||||
| Constraint | Reason |
|
||||
| --- | --- |
|
||||
| TP shard boundaries must align to GGML blocks | Column-parallel rows shard directly; row-parallel packed columns require each local input partition to contain whole quantization blocks |
|
||||
| No `--use-fsdp-inference` | FSDP does not preserve the GGUF packed-block layout |
|
||||
| CUDA only | The reused SRT GGML dequantization kernel currently ships for CUDA |
|
||||
| Native byte order only | A quantized block embeds its scales, so a non-native file cannot be byte-swapped as a whole |
|
||||
| No LoRA, and none of the H3 AdaLN cache flags | An adapter cannot be merged into packed blocks, and the AdaLN paths read the transformer's safetensors |
|
||||
| No `--quantization` | The checkpoint fixes the quantization; the flag would be a second, conflicting selector |
|
||||
|
||||
Every constraint above fails at startup with an explanatory error rather than
|
||||
silently producing wrong output.
|
||||
|
||||
Sequence parallelism (`--ulysses-degree` / `--ring-degree`) remains available
|
||||
because it shards activations rather than packed weights. TP is also available;
|
||||
startup rejects a degree that cuts a row-parallel matrix inside a GGML block.
|
||||
|
||||
### Validated scope
|
||||
|
||||
| Model | Checkpoint | Hardware | Result |
|
||||
| --- | --- | --- | --- |
|
||||
| MiniMax-H3 `fl2va` | [`leejet/MiniMax-H3-GGUF`](https://huggingface.co/leejet/MiniMax-H3-GGUF) `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x RTX 5090 (32 GiB) | t2va 1344x768, 107 frames, video + audio; 19.6-26.3 GiB peak depending on VAE offload |
|
||||
| MiniMax-H3 `fl2va` | `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 10.13 s and 17.17 GiB peak |
|
||||
| MiniMax-H3 `fl2va`, pruned AdaLN curve | [`unsloth/MiniMax-H3-GGUF`](https://huggingface.co/unsloth/MiniMax-H3-GGUF) `minimax_h3_fl2va_pruned-Q4_K.gguf` (10.7 GiB loaded DiT) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC |
|
||||
| MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 1x GB300 (CUDA 13, PyTorch 2.13) | 50-step t2va 1344x768, 107 frames, H.264 + AAC; 105.38 s and 80.88 GB peak |
|
||||
| MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 2x GB300, TP2 (CUDA 13, PyTorch 2.13) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 7.55 s and 51.90 GB peak per rank |
|
||||
|
||||
The DiT loads at 17.5 GiB against 61.7 GiB for the BF16 checkpoint. Weight
|
||||
fidelity was checked tensor-by-tensor against the BF16 reference: cosine
|
||||
1.00000 for the F32/BF16 tensors and 0.9973 for Q4_K/Q4_0.
|
||||
|
||||
Not validated in the measurements above: any other quantization type, the
|
||||
`ref2va` partition, and a BF16-vs-GGUF output comparison.
|
||||
|
||||
## Nunchaku (SVDQuant)
|
||||
|
||||
### Install
|
||||
|
||||
Reference in New Issue
Block a user