[diffusion] feat: load quantized H3 text encoder checkpoints (#34986)

Co-authored-by: Yiqi Yang <yangyiqi8787@gmail.com>
This commit is contained in:
Mick
2026-08-18 09:10:54 +08:00
committed by GitHub
co-authored by Yiqi Yang
parent 0ea262e6e5
commit d55f1c28e2
10 changed files with 351 additions and 5 deletions
@@ -738,6 +738,29 @@ The picker exposes this option only on the B200 and B300 topologies used for
real H3 validation runs.
</Warning>
The Qwen3-VL text encoder can be replaced independently of the DiT. To reduce
its resident memory, point the text-encoder component at the serialized FP8
checkpoint used in validation:
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-H3 \
--model-variant fl2va \
--component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8 \
--num-gpus 4 \
--tp-size 2 \
--ulysses-degree 2 \
--performance-mode speed \
--port 30010
```
`--text-encoder-path` is accepted as a shorter alias. No separate quantization
flag is required: SGLang reads the checkpoint's `quantization_config` and
fails closed if the native encoder does not support that format. The language
linear layers use FP8 while embeddings, normalization, and the vision tower
remain BF16. This is an approximate serve-time choice and is incompatible with
the strict `quality="high"` deployment contract.
</Tab>
</Tabs>
+6
View File
@@ -129,6 +129,12 @@ For quantized transformer checkpoints, prefer:
- `--quantization` for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically)
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
Component checkpoint paths are selected separately, so changing DiT precision
never silently changes prompt embeddings. For a native text encoder:
- `--component-paths.text_encoder {MODEL}` replaces the text-encoder checkpoint; `--text-encoder-path {MODEL}` is its shorter alias
- Quantization metadata is auto-detected from that checkpoint. Each native encoder must explicitly support the serialized format; this is not blanket quantization support for every component, and unsupported combinations fail before weight loading.
For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}`
compresses completed KV-cache chunks independently of transformer weight
quantization. It is lossy and disabled by default.
+27 -3
View File
@@ -2,11 +2,12 @@
title: "Quantization"
tag: "approx"
metatags:
description: "Configure transformer weight quantization and Quant-VideoGen causal KV-cache quantization in SGLang-Diffusion."
description: "Configure transformer and native text-encoder checkpoint quantization, plus Quant-VideoGen causal KV-cache quantization in SGLang-Diffusion."
---
SGLang-Diffusion supports quantized transformer checkpoints. In most cases, keep
the base model and the quantized transformer override separate.
SGLang-Diffusion supports quantized transformer checkpoints and selected
quantized native text-encoder checkpoints. Transformer and text-encoder
precision are resolved independently.
## Quick Reference
@@ -17,6 +18,8 @@ Use these paths:
- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID
- `--quantization`: apply online quantization to unquantized models at load time (activations are quantized dynamically)
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
- `--component-paths.text_encoder`: replace a native text encoder with a checkpoint whose `quantization_config` is auto-detected
- `--text-encoder-path`: shorter alias for `--component-paths.text_encoder`
- `--kv-cache-quant`: compress completed causal KV-cache chunks for supported realtime models
Recommended example for pre-quantized checkpoints:
@@ -280,6 +283,27 @@ sglang generate \
Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantizd if its prefix contains any of the given patterns.
## MiniMax-H3 Text Encoder FP8
MiniMax-H3 can load a serialized FP8 checkpoint for the language linear layers
in its native Qwen3-VL text encoder independently of the DiT. Embeddings,
normalization layers, and the Qwen vision tower remain in BF16.
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-H3 \
--model-variant fl2va \
--component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8 \
--num-gpus 4 \
--port 30010
```
`--text-encoder-path` is accepted as a shorter alias. No quantization flag is
needed: SGLang detects the checkpoint metadata and only enables formats that
the native encoder explicitly supports. Text-encoder FP8 is approximate, is
not enabled by default, and is rejected by MiniMax-H3's strict
`quality="high"` deployment contract.
## Validated ModelOpt Checkpoints
This section is the canonical support matrix for the thirteen published