[diffusion] feat: load quantized H3 text encoder checkpoints (#34986)
Co-authored-by: Yiqi Yang <yangyiqi8787@gmail.com>
This commit is contained in:
@@ -738,6 +738,29 @@ The picker exposes this option only on the B200 and B300 topologies used for
|
||||
real H3 validation runs.
|
||||
</Warning>
|
||||
|
||||
The Qwen3-VL text encoder can be replaced independently of the DiT. To reduce
|
||||
its resident memory, point the text-encoder component at the serialized FP8
|
||||
checkpoint used in validation:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-H3 \
|
||||
--model-variant fl2va \
|
||||
--component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8 \
|
||||
--num-gpus 4 \
|
||||
--tp-size 2 \
|
||||
--ulysses-degree 2 \
|
||||
--performance-mode speed \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
`--text-encoder-path` is accepted as a shorter alias. No separate quantization
|
||||
flag is required: SGLang reads the checkpoint's `quantization_config` and
|
||||
fails closed if the native encoder does not support that format. The language
|
||||
linear layers use FP8 while embeddings, normalization, and the vision tower
|
||||
remain BF16. This is an approximate serve-time choice and is incompatible with
|
||||
the strict `quality="high"` deployment contract.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
@@ -129,6 +129,12 @@ For quantized transformer checkpoints, prefer:
|
||||
- `--quantization` for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically)
|
||||
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
|
||||
|
||||
Component checkpoint paths are selected separately, so changing DiT precision
|
||||
never silently changes prompt embeddings. For a native text encoder:
|
||||
|
||||
- `--component-paths.text_encoder {MODEL}` replaces the text-encoder checkpoint; `--text-encoder-path {MODEL}` is its shorter alias
|
||||
- Quantization metadata is auto-detected from that checkpoint. Each native encoder must explicitly support the serialized format; this is not blanket quantization support for every component, and unsupported combinations fail before weight loading.
|
||||
|
||||
For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}`
|
||||
compresses completed KV-cache chunks independently of transformer weight
|
||||
quantization. It is lossy and disabled by default.
|
||||
|
||||
@@ -2,11 +2,12 @@
|
||||
title: "Quantization"
|
||||
tag: "approx"
|
||||
metatags:
|
||||
description: "Configure transformer weight quantization and Quant-VideoGen causal KV-cache quantization in SGLang-Diffusion."
|
||||
description: "Configure transformer and native text-encoder checkpoint quantization, plus Quant-VideoGen causal KV-cache quantization in SGLang-Diffusion."
|
||||
---
|
||||
|
||||
SGLang-Diffusion supports quantized transformer checkpoints. In most cases, keep
|
||||
the base model and the quantized transformer override separate.
|
||||
SGLang-Diffusion supports quantized transformer checkpoints and selected
|
||||
quantized native text-encoder checkpoints. Transformer and text-encoder
|
||||
precision are resolved independently.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
@@ -17,6 +18,8 @@ Use these paths:
|
||||
- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID
|
||||
- `--quantization`: apply online quantization to unquantized models at load time (activations are quantized dynamically)
|
||||
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
|
||||
- `--component-paths.text_encoder`: replace a native text encoder with a checkpoint whose `quantization_config` is auto-detected
|
||||
- `--text-encoder-path`: shorter alias for `--component-paths.text_encoder`
|
||||
- `--kv-cache-quant`: compress completed causal KV-cache chunks for supported realtime models
|
||||
|
||||
Recommended example for pre-quantized checkpoints:
|
||||
@@ -280,6 +283,27 @@ sglang generate \
|
||||
|
||||
Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantizd if its prefix contains any of the given patterns.
|
||||
|
||||
## MiniMax-H3 Text Encoder FP8
|
||||
|
||||
MiniMax-H3 can load a serialized FP8 checkpoint for the language linear layers
|
||||
in its native Qwen3-VL text encoder independently of the DiT. Embeddings,
|
||||
normalization layers, and the Qwen vision tower remain in BF16.
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-H3 \
|
||||
--model-variant fl2va \
|
||||
--component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8 \
|
||||
--num-gpus 4 \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
`--text-encoder-path` is accepted as a shorter alias. No quantization flag is
|
||||
needed: SGLang detects the checkpoint metadata and only enables formats that
|
||||
the native encoder explicitly supports. Text-encoder FP8 is approximate, is
|
||||
not enabled by default, and is rejected by MiniMax-H3's strict
|
||||
`quality="high"` deployment contract.
|
||||
|
||||
## Validated ModelOpt Checkpoints
|
||||
|
||||
This section is the canonical support matrix for the thirteen published
|
||||
|
||||
Reference in New Issue
Block a user