minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[diffusion] feat: support component-scoped quantization overrides (#36035)

Browse Source
This commit is contained in:
Mick
2026-08-25 09:20:47 +08:00
committed by GitHub
parent ddea7b9156
commit f8f9226cd2
8 changed files with 124 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/sglang-diffusion/api/cli.mdx
+3
View File
@@ -131,6 +131,8 @@ pipeline's registered module name:
| --- | --- | --- | --- |
| Replace a component | `--component-paths.<component> {MODEL}` | `--<component>-path {MODEL}` | Load the replacement component's configuration and weights |
| Replace only its weights | `--component-weights-paths.<component> {WEIGHTS}` | `--<component>-weights-path {WEIGHTS}` | Retain the base component configuration and replace its weights |
| Quantize an unquantized component online | `--component-quantizations.<component> {METHOD}` | `--<component>-quantization {METHOD}` | Apply a method supported by that component's native loader |
| Keep selected component layers unquantized | `--component-quantization-ignored-layers.<component> {PATTERN...}` | None | Pass component-local ignored-layer patterns to its online quantizer |
For example, pair a replacement text-encoder configuration with a separate
single-file checkpoint as follows:
@@ -152,6 +154,7 @@ unquantized during online quantization.
For a native text encoder:
- `--component-paths.text_encoder {MODEL}` replaces the text-encoder checkpoint; `--text-encoder-path {MODEL}` is its shorter alias
- `--component-quantizations.text_encoder {METHOD}` applies supported online quantization to an unquantized native encoder; pair it with `--component-quantization-ignored-layers.text_encoder {PATTERN...}` when selected layers must remain unquantized
- Quantization metadata is auto-detected from that checkpoint. The native loader accepts compatible serialized formats without a model-name allowlist and rejects implementations that do not construct the required quantized layers.
The same contract applies to every weighted component: path routing is generic,
docs/docs/sglang-diffusion/quantization.mdx
+11 -7
View File
@@ -23,6 +23,8 @@ Use these paths:
(local or Hub)
- `--quantization`: override the quantization method used by the transformer loader
- `--quantization-ignored-layers`: transformer layer name patterns to keep unquantized during online quantization (e.g. `attention.to_`)
- `--component-quantizations.<component>`: explicitly apply supported online quantization to an unquantized component
- `--component-quantization-ignored-layers.<component>`: component-local layer patterns to keep unquantized during that online quantization
- `--component-paths.text_encoder`: replace a native text encoder with a checkpoint whose `quantization_config` is auto-detected
- `--text-encoder-path`: shorter alias for `--component-paths.text_encoder`
- `--kv-cache-quant`: compress completed causal KV-cache chunks for supported realtime models
@@ -78,13 +80,15 @@ paths:
| Library-managed Transformers or Diffusers components | Delegates to the upstream `from_pretrained` path and inherits its format support and validation behavior. The local PE model uses this path; compatible formats remain model-specific. |
| Native auxiliary components that load raw state dicts | Quantized checkpoints are rejected before model construction until that component has a quantized materialization implementation. This includes connectors, duration heads, bridges, diffusion decoders, sound tokenizers, spatial upsamplers, and vocoders. |
`--quantization` is the explicit override for the transformer loader; it is not
the boundary of component quantization support. Other pre-quantized component
repositories select their format through their own metadata and the capability
of the selected loader. A generic string override without a matching
materialization backend would advertise support that the component does not
have, while a quantized weight file without matching config metadata cannot be
identified or restored generically.
`--quantization` is the explicit override for the primary transformer loader;
`--component-quantizations.<component>` expresses the same intent for one
supported component. Pair the latter with
`--component-quantization-ignored-layers.<component>` to keep matching layers
unquantized. Pre-quantized component repositories instead select their format
through their own metadata and the capability of the selected loader. A generic
string override without a matching materialization backend would advertise
support that the component does not have, while a quantized weight file without
matching config metadata cannot be identified or restored generically.
## Quant Families