minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support (#22574)

Browse Source
This commit is contained in:
Xiaoyu Zhang
2026-04-13 07:57:41 +08:00
committed by GitHub
parent f81b6df3a3
commit 03a1a7b81c
11 changed files with 872 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/diffusion/quantization.md
+21 -1
View File
@@ -48,6 +48,26 @@ backend.
| `nunchaku-svdq` | Pre-quantized Nunchaku transformer weights, usually named `svdq-{int4\|fp4}_r{rank}-...` | `--transformer-weights-path` | Model-specific support such as Qwen-Image, FLUX, and Z-Image | `nunchaku` | SGLang can infer precision and rank from the filename and supports both `int4` and `nvfp4` |
| `msmodelslim` | Pre-quantized msmodelslim transformer weights | `--model-path` | Wan2.2 family | None | Currently only compatible with the Ascend NPU family and supports both `w8a8` and `w4a4` |
## Validated ModelOpt Checkpoints
This section is the canonical support matrix for diffusion ModelOpt checkpoints
that have been brought up and verified in SGLang.
### FP8
| Base Model | Validated Scope | HF DiT Weights | Notes |
| --- | --- | --- | --- |
| `black-forest-labs/FLUX.1-dev` | single-transformer override, deterministic latent/image comparison, H100 benchmark, torch-profiler trace | `BBuf/flux1-dev-modelopt-fp8-sglang-transformer` | SGLang converter keeps a validated BF16 fallback set for modulation and FF projection layers; use `--model-id FLUX.1-dev` for local mirrors |
| `black-forest-labs/FLUX.2-dev` | single-transformer override load and generation path | `BBuf/flux2-dev-modelopt-fp8-sglang-transformer` | published SGLang-ready transformer override |
| `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | primary `transformer` quantized, `transformer_2` kept BF16 | `BBuf/wan22-t2v-a14b-modelopt-fp8-sglang-transformer` | do not describe this as dual-transformer full-model FP8 unless that path is validated separately |
### NVFP4
| Base Model | Validated Scope | HF DiT Weights | Notes |
| --- | --- | --- | --- |
| `black-forest-labs/FLUX.1-dev` | mixed BF16+NVFP4 transformer override, correctness validation, 4x RTX 5090 benchmark, torch-profiler trace | `unpublished` | use `build_modelopt_nvfp4_transformer.py`; validated builder keeps selected FLUX.1 modules in BF16 and sets `swap_weight_nibbles=false` |
| `black-forest-labs/FLUX.2-dev` | packed-QKV load path | `black-forest-labs/FLUX.2-dev-NVFP4` | validated packed export detection and runtime layout handling |
## ModelOpt FP8
### Usage Examples
@@ -82,7 +102,7 @@ sglang generate \
ModelOpt FP8 checkpoints because the runtime expects the transformed FP8
weights to remain GPU-resident in their column-major layout.
- To build the converted checkpoint yourself from a ModelOpt diffusers export,
use `python -m sglang.multimodal_gen.tools.convert_modelopt_fp8_checkpoint`.
use `python -m sglang.multimodal_gen.tools.build_modelopt_fp8_transformer`.
## NVFP4