[diffusion] feat: support loading minimax h3 gguf text encoders (#36055)

This commit is contained in:
Mick
2026-08-24 23:00:49 +08:00
committed by GitHub
parent 9b0007ed19
commit 30f9ed09d1
10 changed files with 367 additions and 44 deletions
@@ -1006,6 +1006,21 @@ storage, then dequantizes only the active matrix for BF16/FP16 linear math. Use
it as a memory option, not as an INT8 throughput claim. The embedded Quanto map
is the selector; adding `--quantization` would describe a different operation.
Qwen3-VL GGUF files use the same component option:
```bash Overlay
--component-paths.text_encoder \
DeepBeepMeep/MiniMax-H3/Qwen3-VL-32B-Instruct/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf
```
The file selects its own Q4/Q6 layout; do not add `--quantization`. SGLang keeps
the aligned vocabulary table and language linears in packed GGML storage. The
smaller Comfy vision tower, whose flattened blocks can cross row and TP shard
boundaries, is restored one matrix at a time to BF16 during loading.
This CUDA-only memory option supports encoder TP and layerwise placement, but
not encoder FSDP. Like the Quanto variant, it is approximate and is not a
throughput claim.
</Tab>
<Tab title="Encoder scheduling">
+8 -8
View File
@@ -162,11 +162,11 @@ backend.
</tr>
<tr>
<td><code>gguf</code></td>
<td>One selected GGUF DiT file</td>
<td><code>--transformer-weights-path</code></td>
<td>MiniMax-H3 original or pruned FL2VA / Ref2VA DiTs</td>
<td>One selected GGUF DiT or native encoder file</td>
<td><code>--transformer-weights-path</code>, or an explicit encoder file through <code>--component-paths.&lt;component&gt;</code></td>
<td>MiniMax-H3 original or pruned FL2VA / Ref2VA DiTs and its Qwen3-VL text encoder</td>
<td>None</td>
<td>CUDA only; auto-detected; supports standard and K-quant GGML types; FSDP and the separate Qwen3-VL text-encoder GGUF files are not supported</td>
<td>CUDA only and auto-detected; supports standard and K-quant GGML types. TP requires aligned GGML blocks; component/layerwise offload is supported and FSDP is not.</td>
</tr>
<tr>
<td><code>comfy-fp8</code></td>
@@ -242,11 +242,11 @@ backend.
</tr>
<tr>
<td><code>gguf</code></td>
<td>A single community <code>.gguf</code> holding the transformer</td>
<td><code>--transformer-weights-path</code></td>
<td>MiniMax-H3 <code>fl2va</code> (original and pruned AdaLN curve)</td>
<td>A single community <code>.gguf</code> holding a transformer or native encoder</td>
<td><code>--transformer-weights-path</code> or <code>--component-paths.&lt;component&gt;</code></td>
<td>MiniMax-H3 <code>fl2va</code> DiT (original and pruned AdaLN curve) and Qwen3-VL text encoder</td>
<td>None</td>
<td>CUDA only, no FSDP. TP requires GGML-aligned shard boundaries. Shrinks the download and the host memory offload pins (17.5 vs 61.7 GiB for H3) rather than peak VRAM, which offload already bounds. Dequantized per use, so it is not faster. See <a href="#gguf">GGUF</a>.</td>
<td>CUDA only, no FSDP. TP requires GGML-aligned shard boundaries. Packed weights reduce checkpoint and resident-weight memory; they are dequantized per use, so this is not a speed claim. See <a href="#gguf">GGUF</a>.</td>
</tr>
</tbody>
</table>