[diffusion] feat: support serialized comfy convrot int8 dits (#35994)
This commit is contained in:
@@ -281,11 +281,13 @@ contract and low resident weight memory, but that part is slower than a fully
|
||||
quantized FP8 GEMM. TP, Ulysses/Ring sequence parallelism, and
|
||||
component/layerwise offload are supported; FSDP inference is rejected.
|
||||
|
||||
The `pruned_int8_convrot` files are detected but remain unsupported. They
|
||||
require online regular-Hadamard ConvRot, dynamic INT8 activation quantization,
|
||||
and a matching W8A8 GEMM. SGLang fails before loading them instead of silently
|
||||
treating their stored INT8 values as ordinary weights. A native ConvRot kernel
|
||||
path should be added and benchmarked separately before these files are accepted.
|
||||
The `pruned_int8_convrot` files use the same override path and are detected
|
||||
automatically. Install `comfy-kitchen`, replace the FP8 filename above with
|
||||
`minimax_h3_fl2va_pruned_int8_convrot.safetensors`, and still omit
|
||||
`--quantization`. SGLang loads their serialized INT8 weights and row scales into
|
||||
the fused ConvRot kernel without requantizing them. TP1/2/4 and sequence or
|
||||
layerwise offload are supported; TP8 violates the checkpoint's 256-element
|
||||
ConvRot group boundary, and FSDP is rejected.
|
||||
|
||||
### Advanced: precomputed AdaLN cache
|
||||
|
||||
@@ -907,7 +909,7 @@ sglang generate \
|
||||
`--attention-backend sol_attn` with
|
||||
`--attention-backend-config dense_backend=sage_attn,dense_steps=10` and
|
||||
`--component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn`.
|
||||
See [Quantization](/docs/sglang-diffusion/quantization#kitchen-int8-online-quantization)
|
||||
See [Quantization](/docs/sglang-diffusion/quantization#kitchen-int8)
|
||||
and [Attention Backends](/docs/sglang-diffusion/attention_backends#sage-then-sol-hybrid).
|
||||
|
||||
<Warning>
|
||||
|
||||
@@ -50,10 +50,11 @@ directory directly as `--model-path`, but that is a compatibility path. If a
|
||||
repo contains multiple candidate checkpoints, pass
|
||||
`--transformer-weights-path` explicitly.
|
||||
|
||||
MiniMax-H3 auto-detects the per-layer metadata in Comfy's
|
||||
`pruned_fp8_scaled` safetensors. Pass one selected FL2VA or Ref2VA file by local
|
||||
path, `owner/repo/path/file.safetensors`, or direct Hugging Face file URL; do
|
||||
not combine it with `--quantization`. MiniMax-H3 GGUF usage is documented in
|
||||
MiniMax-H3 is a verified example for Comfy safetensors with per-layer metadata,
|
||||
including `pruned_fp8_scaled` and serialized ConvRot INT8. Pass one selected
|
||||
FL2VA or Ref2VA DiT file by local path, `owner/repo/path/file.safetensors`, or
|
||||
direct Hugging Face file URL; do not combine it with `--quantization`. Its GGUF
|
||||
usage is documented in
|
||||
the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#pre-quantized-gguf-transformer).
|
||||
|
||||
## Quantized Component Repositories
|
||||
@@ -165,6 +166,14 @@ backend.
|
||||
<td>None</td>
|
||||
<td>CUDA; auto-detected; TP, sequence parallelism, and component/layerwise offload are supported, while FSDP is not. Checkpoint-marked <code>fc2</code> layers retain FP8 storage and use compute-dtype matmul.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>comfy-int8-convrot</code></td>
|
||||
<td>One selected safetensors file with per-layer <code>int8_tensorwise</code> and ConvRot metadata</td>
|
||||
<td><code>--transformer-weights-path</code></td>
|
||||
<td>MiniMax-H3 native DiT; pruned FL2VA is E2E-verified and Ref2VA has the same validated tensor contract</td>
|
||||
<td><code>comfy-kitchen</code></td>
|
||||
<td>CUDA; auto-detected; uses the fused Kitchen INT8 kernel and validates weight/scale layout before model construction. TP requires every row-parallel input shard to preserve the checkpoint's ConvRot group boundary; the H3 256-group checkpoint supports TP1/2/4, not TP8. Offload is supported; FSDP is not.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>qvg-kv</code></td>
|
||||
<td>Unquantized model with runtime causal KV-cache compression</td>
|
||||
@@ -334,7 +343,14 @@ sglang generate \
|
||||
```
|
||||
**Note:** Requires `aiter` package with MXFP4 kernel support
|
||||
|
||||
### Kitchen INT8 Online Quantization
|
||||
### Kitchen INT8
|
||||
|
||||
Serialized Comfy ConvRot INT8 DiTs are selected through
|
||||
`--transformer-weights-path` and auto-detected from their per-layer markers.
|
||||
They load INT8 weights and row scales directly; omit `--quantization`.
|
||||
|
||||
For a BF16 checkpoint, `--quantization kitchen_int8` instead performs online
|
||||
quantization after loading:
|
||||
|
||||
`kitchen_int8` quantizes DiT linear weights online from the stock BF16
|
||||
checkpoint. Forward uses the fused `comfy_kitchen.int8_linear` op (rotation,
|
||||
|
||||
Reference in New Issue
Block a user