minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[diffusion] doc: update quantization.md (#21356)

Browse Source
This commit is contained in:
Mick
2026-03-25 14:48:38 +08:00
committed by GitHub
parent 6b0f1e3b43
commit 6cc5717e8a
5 changed files with 188 additions and 201 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/diffusion/api/cli.md
+11
View File
@@ -9,6 +9,7 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
## Supported Arguments
### Server Arguments
- `--model-path {MODEL_PATH}`: Path to the model or model ID
@@ -24,6 +25,16 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
- `--cache-dit-config {PATH}`: Path to a Cache-DiT YAML/JSON config (diffusers backend only)
- `--dit-precision {DTYPE}`: Precision for the DiT model (currently supports fp32, fp16, and bf16).
### Quantized Transformers
For quantized transformer checkpoints, prefer:
- `--model-path` for the base model (the pipeline)
- `--transformer-path` for a quantized `transformers` transformer component folder
- `--transformer-weights-path` for a quantized safetensors file, directory, or repo
See [Quantization](../quantization.md) for the supported quantization families and examples.
### Sampling Parameters
docs/diffusion/index.md
+1
View File
@@ -54,6 +54,7 @@ sglang serve --model-path Qwen/Qwen-Image --port 30010
### Usage
- **[CLI Documentation](api/cli.md)** - Command-line interface for `sglang generate` and `sglang serve`
- **[Quantization](quantization.md)** - Quantized transformer checkpoint usage and supported quantization families
- **[OpenAI API](api/openai_api.md)** - OpenAI-compatible API for image/video generation and LoRA management
- **[Post-Processing](api/post_processing.md)** - Frame interpolation (RIFE) and upscaling (Real-ESRGAN)
docs/diffusion/quantization.md
+175
View File
@@ -0,0 +1,175 @@
# Quantization
SGLang-Diffusion supports quantized transformer checkpoints. In most cases, keep
the base model and the quantized transformer override separate.
## Quick Reference
Use these paths:
- `--model-path`: the base or original model
- `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json`
- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID
Recommended example:
```bash
sglang generate \
--model-path black-forest-labs/FLUX.2-dev \
--transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
--prompt "a curious pikachu"
```
For quantized transformers-style transformer component folders:
```bash
sglang generate \
--model-path /path/to/base-model \
--transformer-path /path/to/quantized-transformer \
--prompt "A Logo With Bold Large Text: SGL Diffusion"
```
NOTE: Some model-specific integrations also accept a quantized repo or local
directory directly as `--model-path`, but that is a compatibility path. If a
repo contains multiple candidate checkpoints, pass
`--transformer-weights-path` explicitly.
## Quant Families
Here, `quant_family` means a checkpoint and loading family with shared CLI
usage and loader behavior. It is not just the numeric precision or a kernel
backend.
| quant_family | checkpoint form | canonical CLI | supported models | extra dependency | platform / notes |
|------------------|--------------------------------------------------------------------------------------------|------------------------------------------------------|--------------------------------------------------------------|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| `fp8` | Quantized transformer component folder, or safetensors with `quantization_config` metadata | `--transformer-path` or `--transformer-weights-path` | ALL | None | Component-folder and single-file flows are both supported |
| `nvfp4-modelopt` | NVFP4 safetensors file, sharded directory, or repo providing transformer weights | `--transformer-weights-path` | FLUX.2 | `comfy-kitchen` optional on Blackwell | Blackwell can use a best-performance kit when available; otherwise SGLang falls back to the generic ModelOpt FP4 path |
| `nunchaku-svdq` | Pre-quantized Nunchaku transformer weights, usually named `svdq-{int4\|fp4}_r{rank}-...` | `--transformer-weights-path` | Model-specific support such as Qwen-Image, FLUX, and Z-Image | `nunchaku` | SGLang can infer precision and rank from the filename and supports both `int4` and `nvfp4` |
## NVFP4
### Usage Examples
Recommended usage keeps the base model and quantized transformer override
separate:
```bash
sglang generate \
--model-path black-forest-labs/FLUX.2-dev \
--transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--save-output
```
SGLang also supports passing the NVFP4 repo or local directory directly as
`--model-path`:
```bash
sglang generate \
--model-path black-forest-labs/FLUX.2-dev-NVFP4 \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--save-output
```
### Notes
- `--transformer-weights-path` is still the canonical CLI for NVFP4
transformer checkpoints.
- Direct `--model-path` loading is a compatibility path for FLUX.2 NVFP4-style
repos or local directories.
- If `--transformer-weights-path` is provided explicitly, it takes precedence
over the compatibility `--model-path` flow.
- For local directories, SGLang first looks for `*-mixed.safetensors`, then
falls back to loading from the directory.
- On Blackwell, `comfy-kitchen` can provide the best-performance path when
available; otherwise SGLang falls back to the generic ModelOpt FP4 path.
## Nunchaku (SVDQuant)
### Install
Install the runtime dependency first:
```bash
pip install nunchaku
```
For platform-specific installation methods and troubleshooting, see the
[Nunchaku installation guide](https://nunchaku.tech/docs/nunchaku/installation/installation.html).
### File Naming and Auto-Detection
For Nunchaku checkpoints, `--model-path` should still point to the original
base model, while `--transformer-weights-path` points to the quantized
transformer weights.
If the basename of `--transformer-weights-path` contains the pattern
`svdq-(int4|fp4)_r{rank}`, SGLang will automatically:
- enable SVDQuant
- infer `--quantization-precision`
- infer `--quantization-rank`
Examples:
| checkpoint name fragment | inferred precision | inferred rank | notes |
|--------------------------|--------------------|---------------|-------|
| `svdq-int4_r32` | `int4` | `32` | Standard INT4 checkpoint |
| `svdq-int4_r128` | `int4` | `128` | Higher-quality INT4 checkpoint |
| `svdq-fp4_r32` | `nvfp4` | `32` | `fp4` in the filename maps to CLI value `nvfp4` |
| `svdq-fp4_r128` | `nvfp4` | `128` | Higher-quality NVFP4 checkpoint |
Common filenames:
| filename | precision | rank | typical use |
|----------|-----------|------|-------------|
| `svdq-int4_r32-qwen-image.safetensors` | `int4` | `32` | Balanced default |
| `svdq-int4_r128-qwen-image.safetensors` | `int4` | `128` | Quality-focused |
| `svdq-fp4_r32-qwen-image.safetensors` | `nvfp4` | `32` | RTX 50-series / NVFP4 path |
| `svdq-fp4_r128-qwen-image.safetensors` | `nvfp4` | `128` | Quality-focused NVFP4 |
| `svdq-int4_r32-qwen-image-lightningv1.0-4steps.safetensors` | `int4` | `32` | Lightning 4-step |
| `svdq-int4_r128-qwen-image-lightningv1.1-8steps.safetensors` | `int4` | `128` | Lightning 8-step |
If your checkpoint name does not follow this convention, pass
`--enable-svdquant`, `--quantization-precision`, and `--quantization-rank`
explicitly.
### Usage Examples
Recommended auto-detected flow:
```bash
sglang generate \
--model-path Qwen/Qwen-Image \
--transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \
--prompt "change the raccoon to a cute cat" \
--attention-backend torch_sdpa \
--save-output
```
Manual override when the filename does not encode the quant settings:
```bash
sglang generate \
--model-path Qwen/Qwen-Image \
--transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \
--enable-svdquant \
--quantization-precision int4 \
--quantization-rank 128 \
--prompt "a beautiful sunset" \
--attention-backend torch_sdpa \
--save-output
```
### Notes
- `--transformer-weights-path` is the canonical flag for Nunchaku checkpoints.
Older config names such as `quantized_model_path` are treated as
compatibility aliases.
- Auto-detection only happens when the checkpoint basename matches
`svdq-(int4|fp4)_r{rank}`.
- The CLI values are `int4` and `nvfp4`. In filenames, the NVFP4 variant is
written as `fp4`.
- Lightning checkpoints usually expect matching `--num-inference-steps`, such
as `4` or `8`.
- Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x)
or SM12x GPUs. Hopper (SM90) is currently rejected.
docs/index.rst
+1
View File
@@ -85,6 +85,7 @@ Its core features include:
diffusion/installation
diffusion/compatibility_matrix
diffusion/api/cli
diffusion/quantization
diffusion/api/openai_api
diffusion/performance/index
diffusion/performance/attention_backends
python/sglang/multimodal_gen/docs/quantization.md
-201
View File
@@ -1,201 +0,0 @@
# Quantization
This document introduces the model quantization schemes supported in SGLang and how to use them to reduce memory usage and accelerate inference.
## Nunchaku (SVDQuant)
### Introduction
**SVDQuant** is a Post-Training Quantization (PTQ) technique for diffusion models that quantizes model weights and activations to 4-bit precision (W4A4) while maintaining high visual quality. This method uses Singular Value Decomposition (SVD) to decompose the weight matrix into low-rank components and residuals, effectively absorbing outliers in activations, making 4-bit quantization possible.
**Nunchaku** is a high-performance inference engine that implements SVDQuant, optimized for low-bit neural networks. It is not Quantization-Aware Training (QAT), but directly quantizes pre-trained models.
Paper: [SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models](https://arxiv.org/abs/2411.05007) (ICLR 2025 Spotlight)
### Key Features
SVDQuant significantly reduces memory usage and accelerates inference while maintaining visual quality:
- **Memory Optimization**: Reduces memory usage by **3.6×** compared to BF16 models.
- **Inference Acceleration**:
- **3.0×** faster than the NF4 (W4A16) baseline on desktop/laptop RTX 4090 GPUs.
- **8.7×** speedup on laptop RTX 4090 by eliminating CPU offloading compared to 16-bit models.
- **3.1×** faster than BF16 and NF4 models on RTX 5090 GPUs with NVFP4.
### Supported Precisions
Nunchaku supports two quantization precisions:
- **INT4**: Standard INT4 quantization, supported on NVIDIA GPUs with Compute Capability 7.0+ (RTX 20 series and above).
- **NVFP4**: FP4 quantization, providing better image quality on newer cards like the RTX 5090.
### Usage
#### 1. Install Nunchaku
```bash
pip install nunchaku
```
For more installation information, please refer to the [Nunchaku Official Documentation](https://nunchaku.tech/docs/nunchaku/installation/installation.html).
#### 2. Download Quantized Models
Nunchaku provides pre-quantized model weights available on Hugging Face:
- [nunchaku-ai/nunchaku-qwen-image](https://huggingface.co/nunchaku-ai/nunchaku-qwen-image)
- [Nunchaku FLUX.1 collection](https://huggingface.co/collections/nunchaku-ai/nunchaku-flux1)
Taking Qwen-Image as an example, several quantized models with different configurations are provided:
| Filename | Precision | Rank | Usage |
|----------|-----------|------|-------|
| `svdq-int4_r32-qwen-image.safetensors` | INT4 | 32 | Standard Version |
| `svdq-int4_r128-qwen-image.safetensors` | INT4 | 128 | High-Quality Version |
| `svdq-fp4_r32-qwen-image.safetensors` | NVFP4 | 32 | RTX 5090 Standard Version |
| `svdq-fp4_r128-qwen-image.safetensors` | NVFP4 | 128 | RTX 5090 High-Quality Version |
| `svdq-int4_r32-qwen-image-lightningv1.0-4steps.safetensors` | INT4 | 32 | Lightning 4-Step Version |
| `svdq-int4_r128-qwen-image-lightningv1.1-8steps.safetensors` | INT4 | 128 | Lightning 8-Step Version |
> **Note**: Higher Rank usually means better image quality, but with slightly increased memory usage and computation.
#### 3. Run Quantized Models
SGLang features **smart auto-detection** for Nunchaku models. In most cases, you only need to provide the path to the quantized weights, and the precision and rank will be automatically inferred from the filename.
**Simplified Command (Recommended):**
```bash
sglang generate \
--model-path Qwen/Qwen-Image \
--prompt "change the raccoon to a cute cat" \
--save-output \
--transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors
```
**Manual Override (If needed):**
If your filename doesn't follow the standard naming convention, or you want to force specific settings:
- `--enable-svdquant`: Manually enable SVDQuant.
- `--quantization-precision`: Set to `int4` or `nvfp4`.
- `--quantization-rank`: Set the SVD rank (e.g., 32, 128).
- `--quantization-act-unsigned` (Optional): Use unsigned activation quantization.
Example with manual overrides:
```bash
sglang generate \
--model-path Qwen/Qwen-Image \
--prompt "a beautiful sunset" \
--enable-svdquant \
--transformer-weights-path /path/to/custom_model.safetensors \
--quantization-precision int4 \
--quantization-rank 128
```
#### 4. Configuration Recommendations
Choose the appropriate configuration based on your hardware and requirements:
| Scenario | Recommended Config | Description |
|----------|-------------------|-------------|
| Standard Use (20/30/40 Series GPU) | INT4 + Rank 32 | Balanced performance and quality |
| Quality Focus (Sufficient VRAM) | INT4 + Rank 128 | Better image quality |
| RTX 5090 Standard Use | NVFP4 + Rank 32 | Utilizes FP4 hardware acceleration |
| RTX 5090 Quality Focus | NVFP4 + Rank 128 | Best image quality |
| Fast Prototyping/Preview | Lightning 4-Step Version | Extremely fast generation, slightly reduced quality |
### Notes
1. Model Path Correspondence: `--model-path` should point to the original non-quantized model (for loading config and tokenizer, etc.), while `--transformer-weights-path` points to the quantized weight file / folder / Huggingface Repo ID.
2. Auto-Detection Requirements: For auto-detection to work, the filename must contain the pattern `svdq-{precision}_r{rank}` (e.g., `svdq-int4_r32`).
3. GPU Compatibility:
- INT4: Supports NVIDIA GPUs with Compute Capability 7.0+ (RTX 20 series and above).
- NVFP4: Optimized mainly for newer cards like the RTX 50 series that support FP4.
4. Lightning Models: When using Lightning versions, adjust `--num-inference-steps` accordingly (usually 4 or 8 steps).
### Custom Model Quantization
If you want to quantize your own models, you can use the [DeepCompressor](https://github.com/mit-han-lab/deepcompressor) tool. For detailed instructions, please refer to the Nunchaku official documentation.
## Quantization
### Usage
#### Option 1: Pre-quantized folder (has `config.json`)
For quantized checkpoints that include a `config.json` with a `quantization_config` field (e.g., models converted via `convert_hf_to_fp8.py`), where the transformer's `config.json` already encodes the `quantization_config`, use the component override:
```bash
sglang generate \
--model-path /path/to/FLUX.1-dev \
--transformer-path /path/to/FLUX.1-dev/transformer-FP8 \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--save-output
```
If you need to convert a model to FP8 format yourself, use the provided conversion script:
```bash
# convert transformer to FP8 with block quantization
python -m sglang.multimodal_gen.tools.convert_hf_to_fp8 \
--model-dir /path/to/FLUX.1-dev/transformer \
--save-dir /path/to/FLUX.1-dev/transformer-FP8 \
--strategy block \
--block-size 128 128
```
#### Option 2: Pre-quantized single-file checkpoint (no `config.json`)
Some providers (e.g., [black-forest-labs/FLUX.2-klein-9b-fp8](https://huggingface.co/black-forest-labs/FLUX.2-klein-9b-fp8)) distribute a single `.safetensors` file without a companion `config.json`. Use `--transformer-weights-path` to point to this file (or HuggingFace repo ID) while keeping `--model-path` for the base model:
```bash
sglang generate \
--model-path black-forest-labs/FLUX.2-klein-9B \
--transformer-weights-path black-forest-labs/FLUX.2-klein-9b-fp8 \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--save-output
```
SGLang-Diffusion will automatically read the `quantization_config` metadata embedded in the safetensors file header (if present). For the quant config to be auto-detected, the file's metadata must contain a JSON-encoded `quantization_config` key with at least a `quant_method` field (e.g. `"fp8"`).
Note: this feature is a WIP
#### Option 3: NVFP4 transformer checkpoint / repo
NVFP4 support is currently for `FLUX.2-dev-NVFP4` style checkpoints.
Recommended usage:
```bash
sglang generate \
--model-path black-forest-labs/FLUX.2-dev \
--transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
--prompt "a curious pikachu"
```
This keeps the CLI semantics aligned with other quantization modes:
SGLang also supports passing the NVFP4 repo or local directory directly as `--model-path`.
In that case, SGLang keeps the user-provided NVFP4 path as the model identity, uses `black-forest-labs/FLUX.2-dev` as the base model for `model_index.json` and non-transformer components, and auto-resolves the quantized transformer weights from the NVFP4 repo or local directory.
Example with direct `--model-path`:
```bash
sglang generate \
--model-path /path/to/FLUX.2-dev-NVFP4 \
--prompt "a curious pikachu"
```
Notes:
- If `--transformer-weights-path` is provided explicitly, it still takes precedence.
- For automatic resolution from a local directory, SGLang looks for `*-mixed.safetensors` first, then falls back to the whole directory.
- On Blackwell, if `comfy-kitchen` is not installed, SGLang falls back to the generic ModelOpt FP4 path and prints a warning.