[Diffusion] modelopt diffusion fp8 support for flux1/flux2 and wan2.2 (#22365)
This commit is contained in:
@@ -43,10 +43,47 @@ backend.
|
||||
| quant_family | checkpoint form | canonical CLI | supported models | extra dependency | platform / notes |
|
||||
|------------------|--------------------------------------------------------------------------------------------|------------------------------------------------------|--------------------------------------------------------------|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
|
||||
| `fp8` | Quantized transformer component folder, or safetensors with `quantization_config` metadata | `--transformer-path` or `--transformer-weights-path` | ALL | None | Component-folder and single-file flows are both supported |
|
||||
| `modelopt-fp8` | Converted ModelOpt FP8 transformer directory or repo with `config.json` | `--transformer-weights-path` | FLUX.2, Wan2.2 | None | Override config is read from the quantized transformer repo; `dit_cpu_offload` and `dit_layerwise_offload` are auto-disabled |
|
||||
| `nvfp4-modelopt` | NVFP4 safetensors file, sharded directory, or repo providing transformer weights | `--transformer-weights-path` | FLUX.2 | `comfy-kitchen` optional on Blackwell | Blackwell can use a best-performance kit when available; otherwise SGLang falls back to the generic ModelOpt FP4 path |
|
||||
| `nunchaku-svdq` | Pre-quantized Nunchaku transformer weights, usually named `svdq-{int4\|fp4}_r{rank}-...` | `--transformer-weights-path` | Model-specific support such as Qwen-Image, FLUX, and Z-Image | `nunchaku` | SGLang can infer precision and rank from the filename and supports both `int4` and `nvfp4` |
|
||||
| `msmodelslim` | Pre-quantized msmodelslim transformer weights | `--model-path` | Wan2.2 family | None | Currently only compatible with the Ascend NPU family and supports both `w8a8` and `w4a4` |
|
||||
|
||||
## ModelOpt FP8
|
||||
|
||||
### Usage Examples
|
||||
|
||||
ModelOpt FP8 checkpoints should be converted into an SGLang-loadable transformer
|
||||
override first, then loaded with the original base model:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path black-forest-labs/FLUX.2-dev \
|
||||
--transformer-weights-path BBuf/flux2-dev-modelopt-fp8-sglang-transformer \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--transformer-weights-path BBuf/wan22-t2v-a14b-modelopt-fp8-sglang-transformer \
|
||||
--prompt "a fox walking through neon rain" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- `--transformer-weights-path` is the canonical flag for converted ModelOpt FP8
|
||||
diffusion checkpoints.
|
||||
- If the override repo or local directory contains its own `config.json`,
|
||||
SGLang reads the quantization config from that override instead of relying on
|
||||
the base model config.
|
||||
- `dit_cpu_offload` and `dit_layerwise_offload` are automatically disabled for
|
||||
ModelOpt FP8 checkpoints because the runtime expects the transformed FP8
|
||||
weights to remain GPU-resident in their column-major layout.
|
||||
- To build the converted checkpoint yourself from a ModelOpt diffusers export,
|
||||
use `python -m sglang.multimodal_gen.tools.convert_modelopt_fp8_checkpoint`.
|
||||
|
||||
## NVFP4
|
||||
|
||||
### Usage Examples
|
||||
|
||||
Reference in New Issue
Block a user