From 6cc5717e8a526bbb72b4fc3365361f4bb5486313 Mon Sep 17 00:00:00 2001 From: Mick Date: Wed, 25 Mar 2026 14:48:38 +0800 Subject: [PATCH] [diffusion] doc: update quantization.md (#21356) --- docs/diffusion/api/cli.md | 11 + docs/diffusion/index.md | 1 + docs/diffusion/quantization.md | 175 +++++++++++++++ docs/index.rst | 1 + .../multimodal_gen/docs/quantization.md | 201 ------------------ 5 files changed, 188 insertions(+), 201 deletions(-) create mode 100644 docs/diffusion/quantization.md delete mode 100644 python/sglang/multimodal_gen/docs/quantization.md diff --git a/docs/diffusion/api/cli.md b/docs/diffusion/api/cli.md index 6af075943..a66711484 100644 --- a/docs/diffusion/api/cli.md +++ b/docs/diffusion/api/cli.md @@ -9,6 +9,7 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f ## Supported Arguments + ### Server Arguments - `--model-path {MODEL_PATH}`: Path to the model or model ID @@ -24,6 +25,16 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f - `--cache-dit-config {PATH}`: Path to a Cache-DiT YAML/JSON config (diffusers backend only) - `--dit-precision {DTYPE}`: Precision for the DiT model (currently supports fp32, fp16, and bf16). +### Quantized Transformers + +For quantized transformer checkpoints, prefer: + +- `--model-path` for the base model (the pipeline) +- `--transformer-path` for a quantized `transformers` transformer component folder +- `--transformer-weights-path` for a quantized safetensors file, directory, or repo + +See [Quantization](../quantization.md) for the supported quantization families and examples. + ### Sampling Parameters diff --git a/docs/diffusion/index.md b/docs/diffusion/index.md index 59162e60e..ca80e8b25 100644 --- a/docs/diffusion/index.md +++ b/docs/diffusion/index.md @@ -54,6 +54,7 @@ sglang serve --model-path Qwen/Qwen-Image --port 30010 ### Usage - **[CLI Documentation](api/cli.md)** - Command-line interface for `sglang generate` and `sglang serve` +- **[Quantization](quantization.md)** - Quantized transformer checkpoint usage and supported quantization families - **[OpenAI API](api/openai_api.md)** - OpenAI-compatible API for image/video generation and LoRA management - **[Post-Processing](api/post_processing.md)** - Frame interpolation (RIFE) and upscaling (Real-ESRGAN) diff --git a/docs/diffusion/quantization.md b/docs/diffusion/quantization.md new file mode 100644 index 000000000..871e78b40 --- /dev/null +++ b/docs/diffusion/quantization.md @@ -0,0 +1,175 @@ +# Quantization + +SGLang-Diffusion supports quantized transformer checkpoints. In most cases, keep +the base model and the quantized transformer override separate. + +## Quick Reference + +Use these paths: + +- `--model-path`: the base or original model +- `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json` +- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID + +Recommended example: + +```bash +sglang generate \ + --model-path black-forest-labs/FLUX.2-dev \ + --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \ + --prompt "a curious pikachu" +``` + +For quantized transformers-style transformer component folders: + +```bash +sglang generate \ + --model-path /path/to/base-model \ + --transformer-path /path/to/quantized-transformer \ + --prompt "A Logo With Bold Large Text: SGL Diffusion" +``` + +NOTE: Some model-specific integrations also accept a quantized repo or local +directory directly as `--model-path`, but that is a compatibility path. If a +repo contains multiple candidate checkpoints, pass +`--transformer-weights-path` explicitly. + +## Quant Families + +Here, `quant_family` means a checkpoint and loading family with shared CLI +usage and loader behavior. It is not just the numeric precision or a kernel +backend. + +| quant_family | checkpoint form | canonical CLI | supported models | extra dependency | platform / notes | +|------------------|--------------------------------------------------------------------------------------------|------------------------------------------------------|--------------------------------------------------------------|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------| +| `fp8` | Quantized transformer component folder, or safetensors with `quantization_config` metadata | `--transformer-path` or `--transformer-weights-path` | ALL | None | Component-folder and single-file flows are both supported | +| `nvfp4-modelopt` | NVFP4 safetensors file, sharded directory, or repo providing transformer weights | `--transformer-weights-path` | FLUX.2 | `comfy-kitchen` optional on Blackwell | Blackwell can use a best-performance kit when available; otherwise SGLang falls back to the generic ModelOpt FP4 path | +| `nunchaku-svdq` | Pre-quantized Nunchaku transformer weights, usually named `svdq-{int4\|fp4}_r{rank}-...` | `--transformer-weights-path` | Model-specific support such as Qwen-Image, FLUX, and Z-Image | `nunchaku` | SGLang can infer precision and rank from the filename and supports both `int4` and `nvfp4` | + +## NVFP4 + +### Usage Examples + +Recommended usage keeps the base model and quantized transformer override +separate: + +```bash +sglang generate \ + --model-path black-forest-labs/FLUX.2-dev \ + --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \ + --prompt "A Logo With Bold Large Text: SGL Diffusion" \ + --save-output +``` + +SGLang also supports passing the NVFP4 repo or local directory directly as +`--model-path`: + +```bash +sglang generate \ + --model-path black-forest-labs/FLUX.2-dev-NVFP4 \ + --prompt "A Logo With Bold Large Text: SGL Diffusion" \ + --save-output +``` + +### Notes + +- `--transformer-weights-path` is still the canonical CLI for NVFP4 + transformer checkpoints. +- Direct `--model-path` loading is a compatibility path for FLUX.2 NVFP4-style + repos or local directories. +- If `--transformer-weights-path` is provided explicitly, it takes precedence + over the compatibility `--model-path` flow. +- For local directories, SGLang first looks for `*-mixed.safetensors`, then + falls back to loading from the directory. +- On Blackwell, `comfy-kitchen` can provide the best-performance path when + available; otherwise SGLang falls back to the generic ModelOpt FP4 path. + +## Nunchaku (SVDQuant) + +### Install + +Install the runtime dependency first: + +```bash +pip install nunchaku +``` + +For platform-specific installation methods and troubleshooting, see the +[Nunchaku installation guide](https://nunchaku.tech/docs/nunchaku/installation/installation.html). + +### File Naming and Auto-Detection + +For Nunchaku checkpoints, `--model-path` should still point to the original +base model, while `--transformer-weights-path` points to the quantized +transformer weights. + +If the basename of `--transformer-weights-path` contains the pattern +`svdq-(int4|fp4)_r{rank}`, SGLang will automatically: +- enable SVDQuant +- infer `--quantization-precision` +- infer `--quantization-rank` + +Examples: + +| checkpoint name fragment | inferred precision | inferred rank | notes | +|--------------------------|--------------------|---------------|-------| +| `svdq-int4_r32` | `int4` | `32` | Standard INT4 checkpoint | +| `svdq-int4_r128` | `int4` | `128` | Higher-quality INT4 checkpoint | +| `svdq-fp4_r32` | `nvfp4` | `32` | `fp4` in the filename maps to CLI value `nvfp4` | +| `svdq-fp4_r128` | `nvfp4` | `128` | Higher-quality NVFP4 checkpoint | + +Common filenames: + +| filename | precision | rank | typical use | +|----------|-----------|------|-------------| +| `svdq-int4_r32-qwen-image.safetensors` | `int4` | `32` | Balanced default | +| `svdq-int4_r128-qwen-image.safetensors` | `int4` | `128` | Quality-focused | +| `svdq-fp4_r32-qwen-image.safetensors` | `nvfp4` | `32` | RTX 50-series / NVFP4 path | +| `svdq-fp4_r128-qwen-image.safetensors` | `nvfp4` | `128` | Quality-focused NVFP4 | +| `svdq-int4_r32-qwen-image-lightningv1.0-4steps.safetensors` | `int4` | `32` | Lightning 4-step | +| `svdq-int4_r128-qwen-image-lightningv1.1-8steps.safetensors` | `int4` | `128` | Lightning 8-step | + +If your checkpoint name does not follow this convention, pass +`--enable-svdquant`, `--quantization-precision`, and `--quantization-rank` +explicitly. + +### Usage Examples + +Recommended auto-detected flow: + +```bash +sglang generate \ + --model-path Qwen/Qwen-Image \ + --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \ + --prompt "change the raccoon to a cute cat" \ + --attention-backend torch_sdpa \ + --save-output +``` + +Manual override when the filename does not encode the quant settings: + +```bash +sglang generate \ + --model-path Qwen/Qwen-Image \ + --transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \ + --enable-svdquant \ + --quantization-precision int4 \ + --quantization-rank 128 \ + --prompt "a beautiful sunset" \ + --attention-backend torch_sdpa \ + --save-output +``` + +### Notes + +- `--transformer-weights-path` is the canonical flag for Nunchaku checkpoints. + Older config names such as `quantized_model_path` are treated as + compatibility aliases. +- Auto-detection only happens when the checkpoint basename matches + `svdq-(int4|fp4)_r{rank}`. +- The CLI values are `int4` and `nvfp4`. In filenames, the NVFP4 variant is + written as `fp4`. +- Lightning checkpoints usually expect matching `--num-inference-steps`, such + as `4` or `8`. +- Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x) + or SM12x GPUs. Hopper (SM90) is currently rejected. diff --git a/docs/index.rst b/docs/index.rst index 665942a48..6be8c0d28 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -85,6 +85,7 @@ Its core features include: diffusion/installation diffusion/compatibility_matrix diffusion/api/cli + diffusion/quantization diffusion/api/openai_api diffusion/performance/index diffusion/performance/attention_backends diff --git a/python/sglang/multimodal_gen/docs/quantization.md b/python/sglang/multimodal_gen/docs/quantization.md deleted file mode 100644 index 51df0dd66..000000000 --- a/python/sglang/multimodal_gen/docs/quantization.md +++ /dev/null @@ -1,201 +0,0 @@ -# Quantization - -This document introduces the model quantization schemes supported in SGLang and how to use them to reduce memory usage and accelerate inference. - -## Nunchaku (SVDQuant) - -### Introduction - -**SVDQuant** is a Post-Training Quantization (PTQ) technique for diffusion models that quantizes model weights and activations to 4-bit precision (W4A4) while maintaining high visual quality. This method uses Singular Value Decomposition (SVD) to decompose the weight matrix into low-rank components and residuals, effectively absorbing outliers in activations, making 4-bit quantization possible. - -**Nunchaku** is a high-performance inference engine that implements SVDQuant, optimized for low-bit neural networks. It is not Quantization-Aware Training (QAT), but directly quantizes pre-trained models. - -Paper: [SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models](https://arxiv.org/abs/2411.05007) (ICLR 2025 Spotlight) - -### Key Features - -SVDQuant significantly reduces memory usage and accelerates inference while maintaining visual quality: - -- **Memory Optimization**: Reduces memory usage by **3.6×** compared to BF16 models. -- **Inference Acceleration**: - - **3.0×** faster than the NF4 (W4A16) baseline on desktop/laptop RTX 4090 GPUs. - - **8.7×** speedup on laptop RTX 4090 by eliminating CPU offloading compared to 16-bit models. - - **3.1×** faster than BF16 and NF4 models on RTX 5090 GPUs with NVFP4. - -### Supported Precisions - -Nunchaku supports two quantization precisions: - -- **INT4**: Standard INT4 quantization, supported on NVIDIA GPUs with Compute Capability 7.0+ (RTX 20 series and above). -- **NVFP4**: FP4 quantization, providing better image quality on newer cards like the RTX 5090. - -### Usage - -#### 1. Install Nunchaku - -```bash -pip install nunchaku -``` - -For more installation information, please refer to the [Nunchaku Official Documentation](https://nunchaku.tech/docs/nunchaku/installation/installation.html). - -#### 2. Download Quantized Models - -Nunchaku provides pre-quantized model weights available on Hugging Face: - -- [nunchaku-ai/nunchaku-qwen-image](https://huggingface.co/nunchaku-ai/nunchaku-qwen-image) -- [Nunchaku FLUX.1 collection](https://huggingface.co/collections/nunchaku-ai/nunchaku-flux1) - -Taking Qwen-Image as an example, several quantized models with different configurations are provided: - -| Filename | Precision | Rank | Usage | -|----------|-----------|------|-------| -| `svdq-int4_r32-qwen-image.safetensors` | INT4 | 32 | Standard Version | -| `svdq-int4_r128-qwen-image.safetensors` | INT4 | 128 | High-Quality Version | -| `svdq-fp4_r32-qwen-image.safetensors` | NVFP4 | 32 | RTX 5090 Standard Version | -| `svdq-fp4_r128-qwen-image.safetensors` | NVFP4 | 128 | RTX 5090 High-Quality Version | -| `svdq-int4_r32-qwen-image-lightningv1.0-4steps.safetensors` | INT4 | 32 | Lightning 4-Step Version | -| `svdq-int4_r128-qwen-image-lightningv1.1-8steps.safetensors` | INT4 | 128 | Lightning 8-Step Version | - -> **Note**: Higher Rank usually means better image quality, but with slightly increased memory usage and computation. - -#### 3. Run Quantized Models - -SGLang features **smart auto-detection** for Nunchaku models. In most cases, you only need to provide the path to the quantized weights, and the precision and rank will be automatically inferred from the filename. - -**Simplified Command (Recommended):** - -```bash -sglang generate \ - --model-path Qwen/Qwen-Image \ - --prompt "change the raccoon to a cute cat" \ - --save-output \ - --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors -``` - -**Manual Override (If needed):** - -If your filename doesn't follow the standard naming convention, or you want to force specific settings: - -- `--enable-svdquant`: Manually enable SVDQuant. -- `--quantization-precision`: Set to `int4` or `nvfp4`. -- `--quantization-rank`: Set the SVD rank (e.g., 32, 128). -- `--quantization-act-unsigned` (Optional): Use unsigned activation quantization. - -Example with manual overrides: - -```bash -sglang generate \ - --model-path Qwen/Qwen-Image \ - --prompt "a beautiful sunset" \ - --enable-svdquant \ - --transformer-weights-path /path/to/custom_model.safetensors \ - --quantization-precision int4 \ - --quantization-rank 128 -``` - -#### 4. Configuration Recommendations - -Choose the appropriate configuration based on your hardware and requirements: - -| Scenario | Recommended Config | Description | -|----------|-------------------|-------------| -| Standard Use (20/30/40 Series GPU) | INT4 + Rank 32 | Balanced performance and quality | -| Quality Focus (Sufficient VRAM) | INT4 + Rank 128 | Better image quality | -| RTX 5090 Standard Use | NVFP4 + Rank 32 | Utilizes FP4 hardware acceleration | -| RTX 5090 Quality Focus | NVFP4 + Rank 128 | Best image quality | -| Fast Prototyping/Preview | Lightning 4-Step Version | Extremely fast generation, slightly reduced quality | - -### Notes - -1. Model Path Correspondence: `--model-path` should point to the original non-quantized model (for loading config and tokenizer, etc.), while `--transformer-weights-path` points to the quantized weight file / folder / Huggingface Repo ID. - -2. Auto-Detection Requirements: For auto-detection to work, the filename must contain the pattern `svdq-{precision}_r{rank}` (e.g., `svdq-int4_r32`). - -3. GPU Compatibility: - - INT4: Supports NVIDIA GPUs with Compute Capability 7.0+ (RTX 20 series and above). - - NVFP4: Optimized mainly for newer cards like the RTX 50 series that support FP4. - -4. Lightning Models: When using Lightning versions, adjust `--num-inference-steps` accordingly (usually 4 or 8 steps). - -### Custom Model Quantization - -If you want to quantize your own models, you can use the [DeepCompressor](https://github.com/mit-han-lab/deepcompressor) tool. For detailed instructions, please refer to the Nunchaku official documentation. - -## Quantization - -### Usage - -#### Option 1: Pre-quantized folder (has `config.json`) - -For quantized checkpoints that include a `config.json` with a `quantization_config` field (e.g., models converted via `convert_hf_to_fp8.py`), where the transformer's `config.json` already encodes the `quantization_config`, use the component override: - -```bash -sglang generate \ - --model-path /path/to/FLUX.1-dev \ - --transformer-path /path/to/FLUX.1-dev/transformer-FP8 \ - --prompt "A Logo With Bold Large Text: SGL Diffusion" \ - --save-output -``` - - -If you need to convert a model to FP8 format yourself, use the provided conversion script: - -```bash -# convert transformer to FP8 with block quantization -python -m sglang.multimodal_gen.tools.convert_hf_to_fp8 \ - --model-dir /path/to/FLUX.1-dev/transformer \ - --save-dir /path/to/FLUX.1-dev/transformer-FP8 \ - --strategy block \ - --block-size 128 128 -``` - -#### Option 2: Pre-quantized single-file checkpoint (no `config.json`) - - - -Some providers (e.g., [black-forest-labs/FLUX.2-klein-9b-fp8](https://huggingface.co/black-forest-labs/FLUX.2-klein-9b-fp8)) distribute a single `.safetensors` file without a companion `config.json`. Use `--transformer-weights-path` to point to this file (or HuggingFace repo ID) while keeping `--model-path` for the base model: - -```bash -sglang generate \ - --model-path black-forest-labs/FLUX.2-klein-9B \ - --transformer-weights-path black-forest-labs/FLUX.2-klein-9b-fp8 \ - --prompt "A Logo With Bold Large Text: SGL Diffusion" \ - --save-output -``` - -SGLang-Diffusion will automatically read the `quantization_config` metadata embedded in the safetensors file header (if present). For the quant config to be auto-detected, the file's metadata must contain a JSON-encoded `quantization_config` key with at least a `quant_method` field (e.g. `"fp8"`). - -Note: this feature is a WIP - -#### Option 3: NVFP4 transformer checkpoint / repo - -NVFP4 support is currently for `FLUX.2-dev-NVFP4` style checkpoints. - -Recommended usage: - -```bash -sglang generate \ - --model-path black-forest-labs/FLUX.2-dev \ - --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \ - --prompt "a curious pikachu" -``` - -This keeps the CLI semantics aligned with other quantization modes: - -SGLang also supports passing the NVFP4 repo or local directory directly as `--model-path`. -In that case, SGLang keeps the user-provided NVFP4 path as the model identity, uses `black-forest-labs/FLUX.2-dev` as the base model for `model_index.json` and non-transformer components, and auto-resolves the quantized transformer weights from the NVFP4 repo or local directory. - -Example with direct `--model-path`: - -```bash -sglang generate \ - --model-path /path/to/FLUX.2-dev-NVFP4 \ - --prompt "a curious pikachu" -``` - -Notes: - -- If `--transformer-weights-path` is provided explicitly, it still takes precedence. -- For automatic resolution from a local directory, SGLang looks for `*-mixed.safetensors` first, then falls back to the whole directory. -- On Blackwell, if `comfy-kitchen` is not installed, SGLang falls back to the generic ModelOpt FP4 path and prints a warning.