minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[diffusion] feat: reject unsupported quantized component checkpoints (#35873)

Browse Source
This commit is contained in:
Mick
2026-08-22 09:33:43 +08:00
committed by GitHub
parent 22dafbcbd9
commit b26695a26e
12 changed files with 297 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/sglang-diffusion/api/cli.mdx
+25 -5
View File
@@ -122,21 +122,31 @@ For frame interpolation and upscaling, see [Post-Processing](./post_processing).
### Quantization
For quantized transformer checkpoints, prefer:
For a pre-quantized main transformer checkpoint, prefer:
- `--model-path` for the base pipeline
- `--transformer-path` for a quantized `transformers` transformer component folder
- `--transformer-weights-path` for a quantized safetensors file, directory,
repo, or a supported GGUF transformer file
- `--quantization` for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically)
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
- `--quantization` to override the quantization method used by the transformer loader
- `--quantization-ignored-layers` transformer layer name patterns to keep unquantized during online quantization (e.g. `attention.to_`)
Component checkpoint paths are selected separately, so changing DiT precision
never silently changes prompt embeddings. For a native text encoder:
never silently changes prompt embeddings. Use
`--component-paths.<component> {MODEL}` or its shorter
`--<component>-path {MODEL}` form for any component key from `model_index.json`
or a native pipeline's registered module name.
For a native text encoder:
- `--component-paths.text_encoder {MODEL}` replaces the text-encoder checkpoint; `--text-encoder-path {MODEL}` is its shorter alias
- Quantization metadata is auto-detected from that checkpoint. Each native encoder must explicitly support the serialized format; this is not blanket quantization support for every component, and unsupported combinations fail before weight loading.
The same contract applies to every weighted component: path routing is generic,
while quantized materialization is capability-based. Native auxiliary loaders
that only understand unquantized state dicts reject quantization metadata before
model construction. See [Quantized Component Repositories](../quantization#quantized-component-repositories)
for the current component matrix.
For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}`
compresses completed KV-cache chunks independently of transformer weight
quantization. It is lossy and disabled by default.
@@ -332,7 +342,17 @@ sglang serve \
--vae-path fal/FLUX.2-Tiny-AutoEncoder
```
The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory.
The component key must match a key in the model's `model_index.json` or a native
pipeline's registered module name. The path must be either a Hugging Face repo
ID or a complete component directory.
Path selection and quantized checkpoint support are separate capabilities. A
pre-quantized component repo must carry its quantization metadata, and the
selected loader must support that serialized format. Native plain-state loaders
fail closed; library-managed components inherit the corresponding Transformers
or Diffusers support. The transformer-specific `--quantization` flag does not
select the format of component checkpoints; their own metadata does. See
[Quantized Component Repositories](../quantization#quantized-component-repositories).
## Component Attention Backend Overrides
docs/docs/sglang-diffusion/compatibility_matrix.mdx
+12 -1
View File
@@ -716,6 +716,13 @@ component directory.
The same overrides can also be provided in config files through
`component_paths.<component>`.
Component path overrides are generic, but quantized checkpoint support is
loader-specific. Native SGLang loaders that only materialize plain state dicts
reject detected quantization metadata before weight loading instead of silently
casting the checkpoint into an unquantized module. Library-managed components
inherit the corresponding Transformers or Diffusers support. See
[Quantized Component Repositories](./quantization#quantized-component-repositories).
### Common Syntax
CLI:
@@ -818,7 +825,7 @@ The table below lists concrete Hugging Face component repos that are already use
### Transformer / DiT
- `--transformer-path` is the standard override for the main denoising transformer.
- For quantized transformers, prefer `--transformer-path` or `--transformer-weights-path`; see `quantization.md`.
- For a pre-quantized main transformer, prefer `--transformer-path` or `--transformer-weights-path`; see [Quantization](./quantization).
- `--video-dit-path` and `--audio-dit-path` are only for pipelines that split denoisers by modality.
### Text Encoders and Preprocessors
@@ -838,6 +845,10 @@ The table below lists concrete Hugging Face component repos that are already use
that component.
2. The override key should match the component name in the pipeline's
`model_index.json` or the native pipeline's registered module name.
3. An override path does not enable quantization by itself. A pre-quantized
component must include metadata for a format supported by its selected
loader. `--quantization` overrides the transformer loader; other component
checkpoints select their serialized format through their own metadata.
## Verified LoRA Examples
docs/docs/sglang-diffusion/index.mdx
+1 -1
View File
@@ -35,7 +35,7 @@ sglang serve --model-path Qwen/Qwen-Image --port 30010
- [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server
- [Performance Overview](/docs/sglang-diffusion/performance-optimization): choose speed, memory, parallelism, caching, and quality-tradeoff levers
- [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT, TeaCache, or Spectrum to reduce denoising cost
- [Quantization](/docs/sglang-diffusion/quantization): configure transformer weight and causal KV-cache quantization
- [Quantization](/docs/sglang-diffusion/quantization): configure component checkpoint and causal KV-cache quantization
- [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models): understand session state, causal caches, and realtime-only controls
- [Contributing](/docs/sglang-diffusion/contributing): contribution workflow, adding new models, and CI perf baselines
docs/docs/sglang-diffusion/quantization.mdx
+46 -11
View File
@@ -2,11 +2,13 @@
title: "Quantization"
tag: "approx"
metatags:
description: "Configure transformer and native text-encoder checkpoint quantization, plus Quant-VideoGen causal KV-cache quantization in SGLang-Diffusion."
description: "Configure quantized component repositories, transformer and native text-encoder checkpoint quantization, plus Quant-VideoGen causal KV-cache quantization in SGLang-Diffusion."
---
SGLang-Diffusion supports quantized transformer checkpoints and selected
quantized native text-encoder checkpoints. Transformer and text-encoder
SGLang-Diffusion treats component path selection and quantized checkpoint
materialization as separate capabilities. Every loaded component can use an
independent checkpoint path, but that checkpoint is quantized only when its
selected loader supports the serialized format. Transformer, encoder, and VAE
precision are resolved independently.
## Quick Reference
@@ -14,12 +16,13 @@ precision are resolved independently.
Use these paths:
- `--model-path`: the base or original model
- `--component-paths.<component>` / `--<component>-path`: replace a component from `model_index.json` or a native registered module with an independent repo or local directory
- `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json`
- `--transformer-weights-path`: replacement transformer weights in safetensors
format (file, directory, or Hub repository/file) or a supported GGUF file
(local or Hub)
- `--quantization`: apply online quantization to unquantized models at load time (activations are quantized dynamically)
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
- `--quantization`: override the quantization method used by the transformer loader
- `--quantization-ignored-layers`: transformer layer name patterns to keep unquantized during online quantization (e.g. `attention.to_`)
- `--component-paths.text_encoder`: replace a native text encoder with a checkpoint whose `quantization_config` is auto-detected
- `--text-encoder-path`: shorter alias for `--component-paths.text_encoder`
- `--kv-cache-quant`: compress completed causal KV-cache chunks for supported realtime models
@@ -53,6 +56,33 @@ path, `owner/repo/path/file.safetensors`, or direct Hugging Face file URL; do
not combine it with `--quantization`. MiniMax-H3 GGUF usage is documented in
the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#pre-quantized-gguf-transformer).
## Quantized Component Repositories
Every loaded component can point to an independent repository, but path routing
does not imply that every loader can materialize every quantization format.
SGLang resolves quantized component checkpoints through one of three explicit
paths:
- load through an SGLang quantization implementation;
- delegate a standard component to Transformers or Diffusers;
- fail closed in a native plain-state loader when the format cannot be restored.
| Component path | Quantized checkpoint behavior |
| --- | --- |
| `transformer`, `transformer_2`, `unconditional_transformer`, `audio_dit`, `video_dit` | Uses the SGLang transformer quantization adapters documented below. |
| `text_encoder*`, `image_encoder*` | Requires the native encoder class to declare support for the detected format. MiniMax-H3 FP8 and model-managed integrations such as Ideogram are supported; unknown combinations fail closed. |
| `vae`, `video_vae`, `audio_vae` | A standard top-level Diffusers `quantization_config` is delegated to `AutoModel.from_pretrained`. Native-only VAEs and nested/compression metadata fail closed. |
| Library-managed Transformers or Diffusers components | Delegates to the upstream `from_pretrained` path and inherits its format support and validation behavior. The local PE model uses this path; compatible formats remain model-specific. |
| Native auxiliary components that load raw state dicts | Quantized checkpoints are rejected before model construction until that component has a quantized materialization implementation. This includes connectors, duration heads, bridges, diffusion decoders, sound tokenizers, spatial upsamplers, and vocoders. |
`--quantization` is the explicit override for the transformer loader; it is not
the boundary of component quantization support. Other pre-quantized component
repositories select their format through their own metadata and the capability
of the selected loader. A generic string override without a matching
materialization backend would advertise support that the component does not
have, while a quantized weight file without matching config metadata cannot be
identified or restored generically.
## Quant Families
Here, `quant_family` means a checkpoint and loading family with shared CLI
@@ -99,7 +129,7 @@ backend.
<td><code>fp8</code> (offline quantization)</td>
<td>Quantized transformer component folder, or safetensors with <code>quantization_config</code> metadata</td>
<td><code>--transformer-path</code> or <code>--transformer-weights-path</code></td>
<td>ALL</td>
<td>Native DiTs whose linear layers support the selected FP8 method; validate quality per model</td>
<td>None</td>
<td>Component-folder and single-file flows are both supported</td>
</tr>
@@ -268,11 +298,15 @@ support LongLive2 pinned sinks, global sinks, or dynamically growing caches.
## Online Quantization
Online quantization applies quantization to unquantized models at load time. This is useful for when pre-quantized checkpoints are not available.
This section describes the online methods currently implemented by the
transformer loader. They are useful when a pre-quantized transformer checkpoint
is not available. Encoders, VAEs, and auxiliary components are independent:
their repositories may still carry serialized quantized weights, which are
restored when the selected component loader supports that format.
### FP8 Online Quantization
Apply FP8 quantization to any unquantized model:
Apply FP8 quantization to a supported unquantized DiT checkpoint:
```bash
sglang generate \
@@ -346,8 +380,9 @@ thresholds with `SGLANG_KITCHEN_INT8_MAX_ROWS` and
### Skipping Layers
By default, online quantization quantizes every linear layer in
the transformer. However, `--quantization-ignored-layers` can be used to keep specific layers in their original precision:
By default, transformer online quantization quantizes every supported linear
layer in that component. However, `--quantization-ignored-layers` can keep
specific transformer layers in their original precision:
```bash
sglang generate \
@@ -365,7 +400,7 @@ sglang generate \
--save-output
```
Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantizd if its prefix contains any of the given patterns.
Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantized if its prefix contains any of the given patterns.
## MiniMax-H3 Text Encoder FP8