[diffusion] feat: support loading self-describing quanto int8 encoders (#36052)

This commit is contained in:
Mick
2026-08-24 16:37:09 +08:00
committed by GitHub
parent 5081ad5d4e
commit adc09a1f63
13 changed files with 536 additions and 43 deletions
@@ -969,6 +969,19 @@ Install `comfy-kitchen>=0.2.27` and omit `--quantization`. SGLang automatically
loads its W4A8 language linears and tensorwise INT8 embedding; the unmarked
vision tower remains BF16.
The same component option accepts a self-describing Quanto qint8 file without
an additional quantization flag:
```bash Overlay
--component-paths.text_encoder \
DeepBeepMeep/MiniMax-H3/Qwen3-VL-32B-Instruct/Qwen3-VL-32B-Instruct-layer50_quanto_bf16_int8.safetensors
```
This variant keeps the declared language and vision linear weights in qint8
storage, then dequantizes only the active matrix for BF16/FP16 linear math. Use
it as a memory option, not as an INT8 throughput claim. The embedded Quanto map
is the selector; adding `--quantization` would describe a different operation.
</Tab>
<Tab title="Encoder scheduling">
@@ -200,6 +200,14 @@ backend.
<td><code>comfy-kitchen&gt;=0.2.27</code></td>
<td>Auto-detected; omit <code>--quantization</code>. Requires SM80+ and validates packed weights, group/channel scales, and optional codebooks before model construction. Mixed encoder files may keep their embedding tensorwise INT8. TP must preserve ConvRot group boundaries; offload is supported and FSDP is not.</td>
</tr>
<tr>
<td><code>quanto-int8</code></td>
<td>One native encoder safetensors file with an embedded Quanto quantization map</td>
<td>An explicit weight file through <code>--component-paths.&lt;component&gt;</code></td>
<td>Native encoders whose mapped linear layers consume every declared qint8 entry; MiniMax-H3's Qwen3-VL encoder is supported</td>
<td>None</td>
<td>Auto-detected weight-only qint8 storage. Each active matrix is dequantized to the compute dtype for the ordinary linear operation, so this reduces stored/resident weight memory rather than promising INT8 GEMM speed. TP and offload are supported; FSDP is not.</td>
</tr>
<tr>
<td><code>qvg-kv</code></td>
<td>Unquantized model with runtime causal KV-cache compression</td>