sglang/docs/docs/sglang-diffusion/quantization.mdx

---
title: "Quantization"
tag: "approx"
metatags:
    description: "Configure quantized component repositories, transformer and native text-encoder checkpoint quantization, plus Quant-VideoGen causal KV-cache quantization in SGLang-Diffusion."
---

SGLang-Diffusion treats component path selection and quantized checkpoint
materialization as separate capabilities. Every loaded component can use an
independent checkpoint path, but that checkpoint is quantized only when its
selected loader supports the serialized format. Transformer, encoder, and VAE
precision are resolved independently.

## Quick Reference

Use these paths:

- `--model-path`: the base or original model
- `--component-paths.<component>` / `--<component>-path`: replace a component from `model_index.json` or a native registered module with an independent repo or local directory
- `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json`
- `--transformer-weights-path`: replacement transformer weights in safetensors
  format (file, directory, or Hub repository/file) or a supported GGUF file
  (local or Hub)
- `--quantization`: override the quantization method used by the transformer loader
- `--quantization-ignored-layers`: transformer layer name patterns to keep unquantized during online quantization (e.g. `attention.to_`)
- `--component-paths.text_encoder`: replace a native text encoder with a checkpoint whose `quantization_config` is auto-detected
- `--text-encoder-path`: shorter alias for `--component-paths.text_encoder`
- `--kv-cache-quant`: compress completed causal KV-cache chunks for supported realtime models

Recommended example for pre-quantized checkpoints:

```bash
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "a curious pikachu"
```

For quantized transformers-style transformer component folders:

```bash
sglang generate \
  --model-path /path/to/base-model \
  --transformer-path /path/to/quantized-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion"
```

NOTE: Some model-specific integrations also accept a quantized repo or local
directory directly as `--model-path`, but that is a compatibility path. If a
repo contains multiple candidate checkpoints, pass
`--transformer-weights-path` explicitly.

MiniMax-H3 auto-detects the per-layer metadata in Comfy's
`pruned_fp8_scaled` safetensors. Pass one selected FL2VA or Ref2VA file by local
path, `owner/repo/path/file.safetensors`, or direct Hugging Face file URL; do
not combine it with `--quantization`. MiniMax-H3 GGUF usage is documented in
the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#pre-quantized-gguf-transformer).

## Quantized Component Repositories

Every loaded component can point to an independent repository, but path routing
does not imply that every loader can materialize every quantization format.
SGLang resolves quantized component checkpoints through one of three explicit
paths:

- load through an SGLang quantization implementation;
- delegate a standard component to Transformers or Diffusers;
- fail closed in a native plain-state loader when the format cannot be restored.

| Component path | Quantized checkpoint behavior |
| --- | --- |
| `transformer`, `transformer_2`, `unconditional_transformer`, `audio_dit`, `video_dit` | Uses the SGLang transformer quantization adapters documented below. |
| `text_encoder*`, `image_encoder*` | Requires the native encoder class to declare support for the detected format. MiniMax-H3 FP8 and model-managed integrations such as Ideogram are supported; unknown combinations fail closed. |
| `vae`, `video_vae`, `audio_vae` | A standard top-level Diffusers `quantization_config` is delegated to `AutoModel.from_pretrained`. Native-only VAEs and nested/compression metadata fail closed. |
| Library-managed Transformers or Diffusers components | Delegates to the upstream `from_pretrained` path and inherits its format support and validation behavior. The local PE model uses this path; compatible formats remain model-specific. |
| Native auxiliary components that load raw state dicts | Quantized checkpoints are rejected before model construction until that component has a quantized materialization implementation. This includes connectors, duration heads, bridges, diffusion decoders, sound tokenizers, spatial upsamplers, and vocoders. |

`--quantization` is the explicit override for the transformer loader; it is not
the boundary of component quantization support. Other pre-quantized component
repositories select their format through their own metadata and the capability
of the selected loader. A generic string override without a matching
materialization backend would advertise support that the component does not
have, while a quantized weight file without matching config metadata cannot be
identified or restored generically.

## Quant Families

Here, `quant_family` means a checkpoint and loading family with shared CLI
usage and loader behavior. It is not just the numeric precision or a kernel
backend.

<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
  <colgroup>
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
  </colgroup>
  <thead>
    <tr>
      <th>quant_family</th>
      <th>checkpoint form</th>
      <th>canonical CLI</th>
      <th>supported models</th>
      <th>extra dependency</th>
      <th>platform / notes</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>fp8</code> / <code>mxfp4</code> (online quantization)</td>
      <td>Unquantized checkpoint (offline via AMD Quark coming soon)</td>
      <td><code>--quantization &#123;fp8,mxfp4&#125;</code></td>
      <td>Z-Image-Turbo (validated), others likely work. More support coming soon.</td>
      <td>MXFP4: <code>aiter</code> on ROCm</td>
      <td>MXFP4 requires ROCm and MI350+ (gfx95x). Weights quantized at load time, activations quantized to <code>fp8</code> / <code>mxfp4</code> dynamically.</td>
    </tr>
    <tr>
      <td><code>kitchen_int8</code> (online quantization)</td>
      <td>Unquantized BF16/FP16 checkpoint</td>
      <td><code>--quantization kitchen_int8</code></td>
      <td>MiniMax-H3 (validated on 1× RTX 4090 24 GB)</td>
      <td><code>comfy-kitchen</code></td>
      <td>Data-free INT8 ConvRot at load time via <code>comfy_kitchen.int8_linear</code>. Layers whose input dim is not divisible by the group size stay in BF16. Requires Turing+ (SM75).</td>
    </tr>
    <tr>
      <td><code>fp8</code> (offline quantization)</td>
      <td>Quantized transformer component folder, or safetensors with <code>quantization_config</code> metadata</td>
      <td><code>--transformer-path</code> or <code>--transformer-weights-path</code></td>
      <td>Native DiTs whose linear layers support the selected FP8 method; validate quality per model</td>
      <td>None</td>
      <td>Component-folder and single-file flows are both supported</td>
    </tr>
    <tr>
      <td><code>modelopt-fp8</code></td>
      <td>Converted ModelOpt FP8 transformer directory or repo with <code>config.json</code></td>
      <td><code>--transformer-path</code></td>
      <td>FLUX.1, FLUX.2, Wan2.2, HunyuanVideo, Qwen Image, Qwen Image Edit</td>
      <td>None</td>
      <td>Serialized config stays <code>quant_method=modelopt</code> with <code>quant_algo=FP8</code>; <code>dit_layerwise_offload</code> is supported and <code>dit_cpu_offload</code> stays disabled</td>
    </tr>
    <tr>
      <td><code>modelopt-nvfp4</code></td>
      <td>Mixed transformer directory/repo with <code>config.json</code>, raw NVFP4 safetensors export/repo, or full ModelOpt Diffusers repo</td>
      <td><code>--transformer-path</code> for mixed overrides; <code>--transformer-weights-path</code> for raw exports; <code>--model-path</code> for full repos</td>
      <td>FLUX.1, FLUX.2, Wan2.2, Qwen Image, Qwen Image 2512, Qwen Image Edit, Qwen Image Edit 2511</td>
      <td>None</td>
      <td>Mixed override repos keep the base model separate; full Qwen Image exports can be loaded directly as <code>--model-path</code>; raw exports such as <code>black-forest-labs/FLUX.2-dev-NVFP4</code> still use the weights-path flow</td>
    </tr>
    <tr>
      <td><code>gguf</code></td>
      <td>One selected GGUF DiT file</td>
      <td><code>--transformer-weights-path</code></td>
      <td>MiniMax-H3 original or pruned FL2VA / Ref2VA DiTs</td>
      <td>None</td>
      <td>CUDA only; auto-detected; supports standard and K-quant GGML types; FSDP and the separate Qwen3-VL text-encoder GGUF files are not supported</td>
    </tr>
    <tr>
      <td><code>comfy-fp8</code></td>
      <td>One selected Comfy safetensors file with per-layer <code>comfy_quant</code> metadata</td>
      <td><code>--transformer-weights-path</code></td>
      <td>MiniMax-H3 pruned FL2VA / Ref2VA DiTs</td>
      <td>None</td>
      <td>CUDA; auto-detected; TP, sequence parallelism, and component/layerwise offload are supported, while FSDP is not. Checkpoint-marked <code>fc2</code> layers retain FP8 storage and use compute-dtype matmul.</td>
    </tr>
    <tr>
      <td><code>qvg-kv</code></td>
      <td>Unquantized model with runtime causal KV-cache compression</td>
      <td><code>--kv-cache-quant &#123;int4,int2&#125;</code></td>
      <td>LingBot World realtime causal path</td>
      <td><code>quant-videogen</code></td>
      <td>CUDA only; compresses completed cache chunks rather than model weights; lossy and disabled by default</td>
    </tr>
    <tr>
      <td><code>nunchaku-svdq</code></td>
      <td>Pre-quantized Nunchaku transformer weights, usually named <code>svdq-&#123;int4\|fp4&#125;_r&#123;rank&#125;-...</code></td>
      <td><code>--transformer-weights-path</code></td>
      <td>Model-specific support such as Qwen-Image, FLUX, and Z-Image</td>
      <td><code>nunchaku</code></td>
      <td>SGLang can infer precision and rank from the filename and supports both <code>int4</code> and <code>nvfp4</code></td>
    </tr>
    <tr>
      <td><code>msmodelslim</code></td>
      <td>Pre-quantized msmodelslim transformer weights</td>
      <td><code>--model-path</code></td>
      <td>Wan2.2 family</td>
      <td>None</td>
      <td>Currently only compatible with the Ascend NPU family and supports <code>mxfp8</code>, <code>mxfp4</code>, <code>w8a8</code>, and <code>w4a4</code></td>
    </tr>
    <tr>
      <td><code>gguf</code></td>
      <td>A single community <code>.gguf</code> holding the transformer</td>
      <td><code>--transformer-weights-path</code></td>
      <td>MiniMax-H3 <code>fl2va</code> (original and pruned AdaLN curve)</td>
      <td>None</td>
      <td>CUDA only, no FSDP. TP requires GGML-aligned shard boundaries. Shrinks the download and the host memory offload pins (17.5 vs 61.7 GiB for H3) rather than peak VRAM, which offload already bounds. Dequantized per use, so it is not faster. See <a href="#gguf">GGUF</a>.</td>
    </tr>
  </tbody>
</table>

## Causal KV-Cache Quantization

Quant-VideoGen KV-cache quantization targets long-running autoregressive video
sessions, where the causal self-attention cache can become comparable to the
model weights. It does not change or quantize the checkpoint weights.

See [Realtime and Causal Video Models](./realtime_models) for the session
lifecycle, supported pipelines, and the distinction between realtime and
request-based causal generation.

Install the optional dependency without allowing its stale Torch requirement to
replace SGLang's pinned Torch version, then enable int4 compression when serving
a supported LingBot World realtime pipeline:

```bash
pip install "sglang[diffusion,diffusion-qvg]"
pip install --no-deps quant-videogen==0.1.0

sglang serve \
  --model-path robbyant/lingbot-world-fast-diffusers \
  --pipeline-class-name LingBotWorldCausalDMDPipeline \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --kv-cache-quant int4 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false
```

### Storage Policy

The current chunk is rewritten at every denoising step, so it remains in BF16.
The newest `--kv-cache-quant-keep-recent` completed chunks also remain in BF16.
Older completed chunks are stable and are packed once with Progressive Residual
Quantization (PRQ); their dense BF16 tensors are then released.

When a transformer layer runs attention, its packed visible chunks are
dequantized and concatenated with the recent BF16 chunks. This creates one
layer's dense attention view at a time instead of keeping dense windows
resident for every transformer layer.

### How PRQ Works

For each K or V vector, PRQ uses k-means to select a centroid, then quantizes
the remaining error:

```text
x = centroid_1 + residual_1
residual_1 = centroid_2 + residual_2
...
x_hat = centroid_1 + centroid_2 + ... + dequantize(low_bit_residual)
```

Each additional stage applies another centroid lookup to the previous stage's
residual. SGLang's default uses one stage, 128 centroids, and an int4 or int2
block-quantized residual. More stages or centroids can reduce reconstruction
error but add codebook storage and packing work.

PRQ is the compression algorithm; selecting older completed chunks is the
runtime storage policy that makes it practical. Stable chunks are compressed
once, while mutable and recent chunks avoid repeated packing and retain higher
precision.

### Quality And Performance

<Warning>
KV-cache quantization is lossy. Disabling it uses the original dense BF16 cache
and is bit-exact with the unmodified path. Enabling int4 or int2 reconstructs an
approximation of K and V, so fixed-seed generated frames are not expected to be
pixel-identical to BF16.
</Warning>

In the initial LingBot measurements, int4 used about 47% of the dense resident
KV-cache memory for a 24-frame window and added about 18% per-chunk latency.
Int2 used about 37% of the dense resident KV-cache memory but introduces more
quantization error. These measurements are configuration-specific; benchmark
memory, latency, temporal consistency, identity stability, and motion quality
on the intended session length. Start with int4 unless capacity requires int2.

The current implementation is limited to the LingBot realtime
sliding-window-and-sink path, including Ulysses sequence sharding. It does not
support LongLive2 pinned sinks, global sinks, or dynamically growing caches.

### Tuning Options

| Option | Default | Effect |
| --- | ---: | --- |
| `--kv-cache-quant {off,int4,int2}` | `off` | Enables QVG KV-cache compression and selects residual precision. |
| `--kv-cache-quant-stages` | `1` | Number of progressive centroid-residual stages. |
| `--kv-cache-quant-centroids` | `128` | Number of k-means centroids per stage. |
| `--kv-cache-quant-block-size` | `64` | Block size used to quantize the final residual. |
| `--kv-cache-quant-iters` | `2` | K-means iterations used while packing a chunk. |
| `--kv-cache-quant-asymmetric` | disabled | Uses asymmetric residual quantization. |
| `--kv-cache-quant-keep-recent` | `1` | Number of newest completed chunks retained in BF16. |
| `--kv-cache-quant-sink {0,1}` | `1` | Whether to quantize completed sink chunks. |
| `--kv-cache-quant-sink-keep` | `0` | Number of leading sink chunks retained in BF16. |

## Online Quantization

This section describes the online methods currently implemented by the
transformer loader. They are useful when a pre-quantized transformer checkpoint
is not available. Encoders, VAEs, and auxiliary components are independent:
their repositories may still carry serialized quantized weights, which are
restored when the selected component loader supports that format.

### FP8 Online Quantization

Apply FP8 quantization to a supported unquantized DiT checkpoint:

```bash
sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --quantization fp8 \
  --prompt "a beautiful sunset" \
  --save-output
```

MiniMax-H3 supports this path while preserving its required FP32 patch,
timestep, and output projections. See the
[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-feature-contracts-and-advanced-recipes)
for its distributed serving recipe.

### MXFP4 Online Quantization

MXFP4 provides aggressive 4-bit compression with online quantization. **Note: Requires ROCm and MI350+ (gfx95x) GPU.**

```bash
sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --quantization mxfp4 \
  --prompt "a beautiful sunset" \
  --save-output
```
**Note:** Requires `aiter` package with MXFP4 kernel support

### Kitchen INT8 Online Quantization

`kitchen_int8` quantizes DiT linear weights online from the stock BF16
checkpoint. Forward uses the fused `comfy_kitchen.int8_linear` op (rotation,
dynamic per-row activation quantization, INT8 GEMM, dequant, and bias).
Install the optional dependency first:

```bash
pip install comfy-kitchen
```

```bash
sglang generate \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --quantization kitchen_int8 \
  --attention-backend fa \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 0 \
  --enable-torch-compile false \
  --prompt "A cat walking on a sunny beach, gentle waves." \
  --save-output
```

Quantization runs after the model's weight loaders, so MiniMax-H3's grouped
`qkv` reorder is already applied. Layers whose input dim is not divisible by
the group size (256) stay in BF16 instead of failing the load; H3's AdaLN
projections take that path.

<Warning>
`kitchen_int8` is approximate and is not a consistency ground-truth mode.
The BF16 path is unchanged when `comfy-kitchen` is not installed. See the
[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-feature-contracts-and-advanced-recipes)
for the 24 GB offload recipe, including why `vae` must stay out of
`--layerwise-offload-components`.
</Warning>

Large-M GEMMs (`rows > 8192` and `out_features >= 8192`) are split by rows so
the fused kernel stays on the data-parallel CUTLASS config. Override the
thresholds with `SGLANG_KITCHEN_INT8_MAX_ROWS` and
`SGLANG_KITCHEN_INT8_MIN_SPLIT_N`.

### Skipping Layers

By default, transformer online quantization quantizes every supported linear
layer in that component. However, `--quantization-ignored-layers` can keep
specific transformer layers in their original precision:

```bash
sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --quantization fp8 \
  --quantization-ignored-layers attention.to_ \
  --prompt "a beautiful sunset" \
  --save-output

sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --quantization mxfp4 \
  --quantization-ignored-layers attention.to_ \
  --prompt "a beautiful sunset" \
  --save-output
```

Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantized if its prefix contains any of the given patterns.

## MiniMax-H3 Text Encoder FP8

MiniMax-H3 can load a serialized FP8 checkpoint for the language linear layers
in its native Qwen3-VL text encoder independently of the DiT. Embeddings,
normalization layers, and the Qwen vision tower remain in BF16.

```bash Command
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8 \
  --num-gpus 4 \
  --port 30010
```

`--text-encoder-path` is accepted as a shorter alias. No quantization flag is
needed: SGLang detects the checkpoint metadata and only enables formats that
the native encoder explicitly supports. Text-encoder FP8 is approximate, is
not enabled by default, and is rejected by MiniMax-H3's strict
`quality="high"` deployment contract.

## Transformers Component BnB4

Model components that already have a native Transformers loading path can load
serialized BitsAndBytes 4-bit checkpoints with a standard top-level
`quantization_config`. Plain checkpoints keep using an available native SGLang
implementation. For example, replace FLUX's T5 component with the official
Diffusers checkpoint:

```bash Command
sglang serve \
  --model-path black-forest-labs/FLUX.1-dev \
  --component-paths.text_encoder_2 \
    diffusers/FLUX.1-dev-bnb-4bit/text_encoder_2
```

The quantized component must stay resident. SGLang rejects component or
layerwise offload, nonstandard metadata locations, and native-only component
fallbacks for this path instead of silently changing the checkpoint contract.
Diffusion DiT components declared under the Diffusers library use the separate
quantization backends documented above.

## Validated ModelOpt Checkpoints

This section is the canonical support matrix for the thirteen published
diffusion ModelOpt checkpoints currently wired up in SGLang docs and validation
coverage.

Published checkpoints keep the serialized quantization config as
`quant_method=modelopt`; the FP8 vs NVFP4 split below is a documentation label
derived from `quant_algo`.

Twelve of the thirteen repos live under `lmsys/*`. The FLUX.2 NVFP4 entry keeps
the official `black-forest-labs/FLUX.2-dev-NVFP4` repo.

<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
  <colgroup>
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
    <col style={{width: "16.67%"}} />
  </colgroup>
  <thead>
    <tr>
      <th>Quant Algo</th>
      <th>Base Model</th>
      <th>Preferred CLI</th>
      <th>HF Repo</th>
      <th>Current Scope</th>
      <th>Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>FP8</code></td>
      <td><code>black-forest-labs/FLUX.1-dev</code></td>
      <td><code>--transformer-path</code></td>
      <td><code>lmsys/flux1-dev-modelopt-fp8-sglang-transformer</code></td>
      <td>single-transformer override, deterministic latent/image comparison, H100 benchmark, torch-profiler trace</td>
      <td>SGLang converter keeps a validated BF16 fallback set for modulation and FF projection layers; use <code>--model-id FLUX.1-dev</code> for local mirrors</td>
    </tr>
    <tr>
      <td><code>FP8</code></td>
      <td><code>black-forest-labs/FLUX.2-dev</code></td>
      <td><code>--transformer-path</code></td>
      <td><code>lmsys/flux2-dev-modelopt-fp8-sglang-transformer</code></td>
      <td>single-transformer override load and generation path</td>
      <td>published SGLang-ready transformer override</td>
    </tr>
    <tr>
      <td><code>FP8</code></td>
      <td><code>Wan-AI/Wan2.2-T2V-A14B-Diffusers</code></td>
      <td><code>--transformer-path</code></td>
      <td><code>lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer</code></td>
      <td>primary <code>transformer</code> quantized, <code>transformer_2</code> kept BF16</td>
      <td>primary-transformer-only path; keep <code>transformer_2</code> on the base checkpoint, and do not describe this as dual-transformer full-model FP8 unless that path is validated separately</td>
    </tr>
    <tr>
      <td><code>FP8</code></td>
      <td><code>hunyuanvideo-community/HunyuanVideo</code></td>
      <td><code>--transformer-path</code></td>
      <td><code>lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer</code></td>
      <td>single-transformer override, BF16-vs-FP8 video comparison, H100 benchmark, torch-profiler trace</td>
      <td>HunyuanVideo uses different ModelOpt/diffusers and SGLang runtime module names; the converter maps those names before writing FP8 scale tensors and BF16 fallback ignores</td>
    </tr>
    <tr>
      <td><code>FP8</code></td>
      <td><code>Qwen/Qwen-Image</code></td>
      <td><code>--transformer-path</code></td>
      <td><code>lmsys/qwen-image-modelopt-fp8-sglang-transformer</code></td>
      <td>single-transformer override, BF16-vs-FP8 image comparison, H100 benchmark, torch-profiler trace</td>
      <td>shares the Qwen Image FP8 fallback preset; keep <code>img_in</code>, <code>txt_in</code>, timestep embedder, <code>norm_out.linear</code>, <code>proj_out</code>, <code>img_mod</code>/<code>txt_mod</code>, and <code>img_mlp.net.2</code> in BF16</td>
    </tr>
    <tr>
      <td><code>FP8</code></td>
      <td><code>Qwen/Qwen-Image-Edit-2511</code></td>
      <td><code>--transformer-path</code></td>
      <td><code>lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer</code></td>
      <td>TI2I edit path, BF16-vs-FP8 image comparison, H100 benchmark</td>
      <td>shares <code>QwenImageTransformer2DModel</code> with Qwen Image and uses the same Qwen Image FP8 fallback preset</td>
    </tr>
    <tr>
      <td><code>NVFP4</code></td>
      <td><code>black-forest-labs/FLUX.1-dev</code></td>
      <td><code>--transformer-path</code></td>
      <td><code>lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer</code></td>
      <td>mixed BF16+NVFP4 transformer override, correctness validation, 4x RTX 5090 benchmark, torch-profiler trace</td>
      <td>use <code>build_modelopt_nvfp4_transformer.py</code>; validated builder keeps selected FLUX.1 modules in BF16 and sets <code>swap_weight_nibbles=false</code></td>
    </tr>
    <tr>
      <td><code>NVFP4</code></td>
      <td><code>black-forest-labs/FLUX.2-dev</code></td>
      <td><code>--transformer-weights-path</code></td>
      <td><code>black-forest-labs/FLUX.2-dev-NVFP4</code></td>
      <td>packed-QKV load path</td>
      <td>official raw export repo; validated packed export detection and runtime layout handling</td>
    </tr>
    <tr>
      <td><code>NVFP4</code></td>
      <td><code>Wan-AI/Wan2.2-T2V-A14B-Diffusers</code></td>
      <td><code>--transformer-path</code></td>
      <td><code>lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer</code></td>
      <td>primary <code>transformer</code> quantized with ModelOpt NVFP4, <code>transformer_2</code> kept BF16</td>
      <td>primary-transformer-only path; keep <code>transformer_2</code> on the base checkpoint; the default FP4 GEMM backend is <code>flashinfer_trtllm</code></td>
    </tr>
    <tr>
      <td><code>NVFP4</code></td>
      <td><code>Qwen/Qwen-Image</code></td>
      <td><code>--model-path</code></td>
      <td><code>lmsys/qwen-image-modelopt-nvfp4-sglang</code></td>
      <td>full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison</td>
      <td>full repo loaded directly; exported with ModelOpt PR #1706 SVDQuant NVFP4 (<code>--format fp4</code>, max calibration, block size 16) and BF16 fallbacks for attention-sensitive modules plus first/last transformer blocks</td>
    </tr>
    <tr>
      <td><code>NVFP4</code></td>
      <td><code>Qwen/Qwen-Image-2512</code></td>
      <td><code>--model-path</code></td>
      <td><code>lmsys/qwen-image-2512-modelopt-nvfp4-sglang</code></td>
      <td>full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison, B200 CI case</td>
      <td>same full-repo loader path as Qwen Image; this is the Qwen Image NVFP4 representative in <code>multimodal-gen-test-1-b200</code></td>
    </tr>
    <tr>
      <td><code>NVFP4</code></td>
      <td><code>Qwen/Qwen-Image-Edit</code></td>
      <td><code>--model-path</code></td>
      <td><code>lmsys/qwen-image-edit-modelopt-nvfp4-sglang</code></td>
      <td>TI2I edit full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison</td>
      <td>full repo loaded directly with normal image-edit inputs; exported with the same ModelOpt PR #1706 NVFP4 recipe</td>
    </tr>
    <tr>
      <td><code>NVFP4</code></td>
      <td><code>Qwen/Qwen-Image-Edit-2511</code></td>
      <td><code>--model-path</code></td>
      <td><code>lmsys/qwen-image-edit-2511-modelopt-nvfp4-sglang</code></td>
      <td>TI2I edit full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison</td>
      <td>full repo loaded directly with normal image-edit inputs; exported with the same ModelOpt PR #1706 NVFP4 recipe</td>
    </tr>
  </tbody>
</table>

These thirteen checkpoints are the intended ModelOpt documentation support
set. The B200 diffusion CI job (`multimodal-gen-test-1-b200`) uses a
representative NVFP4 subset and includes
`lmsys/qwen-image-2512-modelopt-nvfp4-sglang` for Qwen Image coverage.

## ModelOpt FP8

### Usage Examples

Converted ModelOpt FP8 transformer repos should be loaded as transformer
component overrides. If the repo or local directory already contains
`config.json`, use `--transformer-path`. Full Diffusers repos such as the
NVIDIA Wan2.2 FP8 checkpoint can be passed directly with `--model-path`.

```bash
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output
```

```bash
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --transformer-path lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer \
  --prompt "a fox walking through neon rain" \
  --save-output
```

```bash
sglang generate \
  --model-path hunyuanvideo-community/HunyuanVideo \
  --transformer-path lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer \
  --height 544 --width 960 --num-frames 17 \
  --prompt "A cinematic shot of a red sports car driving through rain at night" \
  --save-output
```

```bash
sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-path lmsys/qwen-image-modelopt-fp8-sglang-transformer \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output
```

```bash
sglang generate \
  --model-path Qwen/Qwen-Image-Edit-2511 \
  --transformer-path lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer \
  --image-path /path/to/input.png \
  --prompt "Turn the scene into a warm watercolor illustration" \
  --save-output
```

### Notes

- `--transformer-path` is the canonical flag for converted ModelOpt FP8
  transformer component repos or directories that already carry `config.json`.
- If the override repo or local directory contains its own `config.json`,
  SGLang reads the quantization config from that override instead of relying on
  the base model config.
- `--transformer-weights-path` still works when you intentionally point at raw
  weight files or a directory that should be metadata-probed as weights first.
- `dit_layerwise_offload` is supported for ModelOpt FP8 checkpoints.
- `dit_cpu_offload` still stays disabled for ModelOpt FP8 checkpoints.
- The layerwise offload path now preserves the non-contiguous FP8 weight stride
  expected by the runtime FP8 GEMM path.
- On disk, the quantization config stays `quant_method=modelopt` with
  `quant_algo=FP8`; the `modelopt-fp8` label in this document is a support
  family name, not a serialized config key.
- To build the converted checkpoint yourself from a ModelOpt diffusers export,
  use `python -m sglang.multimodal_gen.tools.build_modelopt_fp8_transformer`.

## ModelOpt NVFP4

### Usage Examples

For mixed ModelOpt NVFP4 transformer overrides that already contain
`config.json`, keep the base model and quantized transformer separate and use
`--transformer-path`:

```bash
sglang generate \
  --model-path black-forest-labs/FLUX.1-dev \
  --transformer-path lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output
```

For raw NVFP4 exports such as the official FLUX.2 release, use
`--transformer-weights-path`:

```bash
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output
```

SGLang also supports passing the NVFP4 repo or local directory directly as
`--model-path`:

```bash
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output
```

For a dual-transformer Wan2.2 export where only the primary `transformer`
was quantized:

```bash
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --transformer-path lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer \
  --prompt "a fox walking through neon rain" \
  --save-output
```

For full Qwen Image NVFP4 exports, load the published repo directly:

```bash
sglang generate \
  --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output
```

For high-resolution Qwen-Image-family generations on B200, the FlashInfer
CUTLASS FP4 GEMM backend can be faster than the default TensorRT-LLM backend:

```bash
SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \
sglang generate \
  --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
  --width 2048 --height 2048 \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output
```

### Notes

- Use `--transformer-path` for mixed ModelOpt NVFP4 transformer repos or local
  directories that already include `config.json`.
- Use `--transformer-weights-path` for raw NVFP4 exports, individual
  safetensors files, or repo layouts that should be treated as weights first.
- For dual-transformer pipelines such as `Wan2.2-T2V-A14B-Diffusers`, the
  primary `--transformer-path` override targets only `transformer`. Use a
  per-component override such as `--transformer-2-path` only when you
  intentionally want a non-default `transformer_2`.
- On Blackwell, the diffusion ModelOpt NVFP4 path defaults to FlashInfer
  TensorRT-LLM FP4 GEMM (`flashinfer_trtllm`).
- The published Qwen Image NVFP4 exports keep the `img_mod`/`txt_mod`
  modulation projections and first/last transformer blocks in BF16.
- Qwen-Image NVFP4 does not always improve latency at 1024x1024. On B200, the
  validated ModelOpt exports were faster than BF16 at 2048x2048 with
  `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass`, while 1024x1024
  remained BF16-faster.
- Direct `--model-path` loading is the canonical path for full Qwen Image
  ModelOpt NVFP4 repos and a compatibility path for FLUX.2 NVFP4-style repos
  or local directories.
- If `--transformer-weights-path` is provided explicitly, it takes precedence
  over the compatibility `--model-path` flow.
- For local directories, SGLang first looks for `*-mixed.safetensors`, then
  falls back to loading from the directory.
- To force the diffusion ModelOpt FP4 path onto a different FlashInfer
  backend, set `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND`. Supported values
  include `flashinfer_cudnn`, `flashinfer_cutlass`, and `flashinfer_trtllm`.
- On disk, the quantization config stays `quant_method=modelopt` with
  `quant_algo=NVFP4`; the `modelopt-nvfp4` label here is again a documentation
  family name rather than a serialized config key.

## GGUF

GGUF loads a community-quantized transformer from a single `.gguf` file while
the rest of the pipeline — VAE, text encoder, scheduler, tokenizer — keeps
loading from the base model.

GGUF primarily reduces checkpoint, host-memory, and resident-weight size. For
example, MiniMax-H3's transformer is 17.5 GiB as Q4_K_M versus 61.7 GiB as
BF16. With full layerwise offload, VAE decode and offload buffers can still
dominate peak GPU memory, but each streamed DiT layer also transfers fewer
bytes.

Packed linears reuse SRT's GGUF type definitions and CUDA dequantization, then
run the native GEMM. SRT's fused MMVQ/MMQ kernels target the low-token LLM
regime and are slower at diffusion sequence lengths. GGUF remains a
capacity-oriented option; latency depends on the quantization type, activation
shape, and placement policy.

Layers the checkpoint stores unquantized (F32/F16/BF16) take the ordinary linear
path rather than the packed one. Their precision is then whatever the model
declares for that layer, exactly as on the safetensors path — a checkpoint
cannot raise a layer above the model's own dtype by storing it wider. For
MiniMax-H3 the two agree: the layers it pins to FP32 are the ones the validated
checkpoint leaves unquantized, and `post_load_weights` fails the load if any of
them ends up narrower.

### Usage

No extra install: `gguf` is already a core SGLang dependency.

`--model-path` stays the base model; `--transformer-weights-path` takes the
GGUF. A local path, `owner/repo/file.gguf`, or `owner/repo:QUANT_TYPE` all work.
The quant-type shorthand is accepted only when exactly one repository file
matches it; otherwise SGLang lists the candidates and asks for a full path.

```bash
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --transformer-weights-path \
    leejet/MiniMax-H3-GGUF/minimax_h3_fl2va-Q4_K_M.gguf \
  --num-gpus 1 \
  --attention-backend fa \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 0 \
  --enable-torch-compile false \
  --port 30010
```

Note that `--quantization gguf` is not the selector — the quantization is read
from the file itself, so passing the file is what enables the path.

MiniMax-H3 GGUF checkpoints may use either the original timestep MLP or the
pruned AdaLN curve architecture. Repositories often contain both FL2VA and
Ref2VA files, so use the full Hub file reference instead of an ambiguous
`owner/repo:QUANT_TYPE` selector:

```bash
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --transformer-weights-path \
    unsloth/MiniMax-H3-GGUF/minimax_h3_fl2va_pruned-Q4_K.gguf \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --enable-torch-compile false \
  --port 30010
```

The pruned architecture keeps its sampled curve and reduced AdaLN projections
in FP32, matching the published checkpoint implementation. This precision
island deliberately bypasses the BF16-only fused modulation kernels.

#### Whether to offload the VAE

Adding `vae` to `--layerwise-offload-components` trades a lot of latency for
some peak VRAM, because the video VAE decoder is re-streamed per decode tile.
Measured on 1× RTX 5090 with this checkpoint, 1344×768 × 107 frames:

| `--layerwise-offload-components` | Peak VRAM | Denoise | VAE decode |
| --- | ---: | ---: | ---: |
| `dit,text_encoder` | 26.3 GiB | 39.0 s | **9.5 s** |
| `dit,text_encoder,vae` | **19.6 GiB** | 39.0 s | 57.2 s |

Denoise is unaffected, and the output is bit-identical either way. Leave the VAE
resident unless the 6.7 GiB matters — on a 24 GB card by this measurement it
does, and the 6× slower decode is the price of fitting.

### Constraints

| Constraint | Reason |
| --- | --- |
| TP shard boundaries must align to GGML blocks | Column-parallel rows shard directly; row-parallel packed columns require each local input partition to contain whole quantization blocks |
| No `--use-fsdp-inference` | FSDP does not preserve the GGUF packed-block layout |
| CUDA only | The reused SRT GGML dequantization kernel currently ships for CUDA |
| Native byte order only | A quantized block embeds its scales, so a non-native file cannot be byte-swapped as a whole |
| No LoRA, and none of the H3 AdaLN cache flags | An adapter cannot be merged into packed blocks, and the AdaLN paths read the transformer's safetensors |
| No `--quantization` | The checkpoint fixes the quantization; the flag would be a second, conflicting selector |

Every constraint above fails at startup with an explanatory error rather than
silently producing wrong output.

Sequence parallelism (`--ulysses-degree` / `--ring-degree`) remains available
because it shards activations rather than packed weights. TP is also available;
startup rejects a degree that cuts a row-parallel matrix inside a GGML block.

### Validated scope

| Model | Checkpoint | Hardware | Result |
| --- | --- | --- | --- |
| MiniMax-H3 `fl2va` | [`leejet/MiniMax-H3-GGUF`](https://huggingface.co/leejet/MiniMax-H3-GGUF) `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x RTX 5090 (32 GiB) | t2va 1344x768, 107 frames, video + audio; 19.6-26.3 GiB peak depending on VAE offload |
| MiniMax-H3 `fl2va` | `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 10.13 s and 17.17 GiB peak |
| MiniMax-H3 `fl2va`, pruned AdaLN curve | [`unsloth/MiniMax-H3-GGUF`](https://huggingface.co/unsloth/MiniMax-H3-GGUF) `minimax_h3_fl2va_pruned-Q4_K.gguf` (10.7 GiB loaded DiT) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC |
| MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 1x GB300 (CUDA 13, PyTorch 2.13) | 50-step t2va 1344x768, 107 frames, H.264 + AAC; 105.38 s and 80.88 GB peak |
| MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 2x GB300, TP2 (CUDA 13, PyTorch 2.13) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 7.55 s and 51.90 GB peak per rank |

The DiT loads at 17.5 GiB against 61.7 GiB for the BF16 checkpoint. Weight
fidelity was checked tensor-by-tensor against the BF16 reference: cosine
1.00000 for the F32/BF16 tensors and 0.9973 for Q4_K/Q4_0.

Not validated in the measurements above: any other quantization type, the
`ref2va` partition, and a BF16-vs-GGUF output comparison.

## Nunchaku (SVDQuant)

### Install

Install the runtime dependency first:

```bash
pip install nunchaku
```

For platform-specific installation methods and troubleshooting, see the
[Nunchaku installation guide](https://nunchaku.tech/docs/nunchaku/installation/installation.html).

### File Naming and Auto-Detection

For Nunchaku checkpoints, `--model-path` should still point to the original
base model, while `--transformer-weights-path` points to the quantized
transformer weights.

If the basename of `--transformer-weights-path` contains the pattern
`svdq-(int4|fp4)_r{rank}`, SGLang will automatically:
- enable SVDQuant
- infer `--quantization-precision`
- infer `--quantization-rank`

Examples:

<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
  <colgroup>
    <col style={{width: "25%"}} />
    <col style={{width: "25%"}} />
    <col style={{width: "25%"}} />
    <col style={{width: "25%"}} />
  </colgroup>
  <thead>
    <tr>
      <th>checkpoint name fragment</th>
      <th>inferred precision</th>
      <th>inferred rank</th>
      <th>notes</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>svdq-int4_r32</code></td>
      <td><code>int4</code></td>
      <td><code>32</code></td>
      <td>Standard INT4 checkpoint</td>
    </tr>
    <tr>
      <td><code>svdq-int4_r128</code></td>
      <td><code>int4</code></td>
      <td><code>128</code></td>
      <td>Higher-quality INT4 checkpoint</td>
    </tr>
    <tr>
      <td><code>svdq-fp4_r32</code></td>
      <td><code>nvfp4</code></td>
      <td><code>32</code></td>
      <td><code>fp4</code> in the filename maps to CLI value <code>nvfp4</code></td>
    </tr>
    <tr>
      <td><code>svdq-fp4_r128</code></td>
      <td><code>nvfp4</code></td>
      <td><code>128</code></td>
      <td>Higher-quality NVFP4 checkpoint</td>
    </tr>
  </tbody>
</table>

Common filenames:

<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
  <colgroup>
    <col style={{width: "25%"}} />
    <col style={{width: "25%"}} />
    <col style={{width: "25%"}} />
    <col style={{width: "25%"}} />
  </colgroup>
  <thead>
    <tr>
      <th>filename</th>
      <th>precision</th>
      <th>rank</th>
      <th>typical use</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code>svdq-int4_r32-qwen-image.safetensors</code></td>
      <td><code>int4</code></td>
      <td><code>32</code></td>
      <td>Balanced default</td>
    </tr>
    <tr>
      <td><code>svdq-int4_r128-qwen-image.safetensors</code></td>
      <td><code>int4</code></td>
      <td><code>128</code></td>
      <td>Quality-focused</td>
    </tr>
    <tr>
      <td><code>svdq-fp4_r32-qwen-image.safetensors</code></td>
      <td><code>nvfp4</code></td>
      <td><code>32</code></td>
      <td>RTX 50-series / NVFP4 path</td>
    </tr>
    <tr>
      <td><code>svdq-fp4_r128-qwen-image.safetensors</code></td>
      <td><code>nvfp4</code></td>
      <td><code>128</code></td>
      <td>Quality-focused NVFP4</td>
    </tr>
    <tr>
      <td><code>svdq-int4_r32-qwen-image-lightningv1.0-4steps.safetensors</code></td>
      <td><code>int4</code></td>
      <td><code>32</code></td>
      <td>Lightning 4-step</td>
    </tr>
    <tr>
      <td><code>svdq-int4_r128-qwen-image-lightningv1.1-8steps.safetensors</code></td>
      <td><code>int4</code></td>
      <td><code>128</code></td>
      <td>Lightning 8-step</td>
    </tr>
  </tbody>
</table>

If your checkpoint name does not follow this convention, pass
`--enable-svdquant`, `--quantization-precision`, and `--quantization-rank`
explicitly.

### Usage Examples

Recommended auto-detected flow:

```bash
sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \
  --prompt "a beautiful sunset" \
  --save-output
```

Manual override when the filename does not encode the quant settings:

```bash
sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \
  --enable-svdquant \
  --quantization-precision int4 \
  --quantization-rank 128 \
  --prompt "a beautiful sunset" \
  --save-output
```

### Notes

- `--transformer-weights-path` is the canonical flag for Nunchaku checkpoints.
  Older config names such as `quantized_model_path` are treated as
  compatibility aliases.
- Auto-detection only happens when the checkpoint basename matches
  `svdq-(int4|fp4)_r{rank}`.
- The CLI values are `int4` and `nvfp4`. In filenames, the NVFP4 variant is
  written as `fp4`.
- Lightning checkpoints usually expect matching `--num-inference-steps`, such
  as `4` or `8`.
- Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x)
  or SM12x GPUs. Hopper (SM90) is currently rejected.

## [ModelSlim](https://gitcode.com/Ascend/msmodelslim)
MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware.

- **Installation**

    ```bash
    # Clone repo and install msmodelslim:
    git clone https://gitcode.com/Ascend/msmodelslim.git
    cd msmodelslim
    bash install.sh
    ```

- **Multimodal_sd quantization**

    Download the original floating-point weights of the large model. Taking Wan2.2-T2V-A14B as an example, you can go to [Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the modelscope model card).
    > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).

  Run quantization using one-click quantization (recommended):

  ```bash
  msmodelslim quant \
    --model_path /path/to/wan2_2_float_weights \
    --save_path /path/to/wan2_2_quantized_weights \
    --device npu \
    --model_type Wan2_2 \
    --quant_type w8a8 \
    --trust_remote_code True
  ```

  For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/README.md) section in ModelSLim repo.

  > Note: SGLang does not support quantized embeddings, please disable this option when quantizing using msmodelslim.

- **Auto-Detection and different formats**

    For msmodelslim checkpoints, it's enough to specify only ```--model-path```, the detection of quantization occurs automatically for each layer using parsing of      `quant_model_description.json` config.

    In the case of `Wan2.2` only `Diffusers` weights storage format are supported, whereas modelslim saves the quantized model in the original `Wan2.2` format.
    For conversion, use the one-step `wan_repack.py` script:

    ```bash
    python wan_repack.py \
      --model-type Wan2.2-TI2V-5B \
      --original-model-path {path_to_original_diffusers_model} \
      --quant-path {path_to_quantized_model} \
      --output-path {path_to_converted_model}
    ```

    Supported `--model-type` values: `Wan2.2-TI2V-5B` (single-transformer), `Wan2.2-T2V-A14B` and `Wan2.2-I2V-A14B` (Cascade dual-transformer).
    The script automatically handles: copying the base model, converting quantized weights to Diffusers format, and restoring `config.json`.

- **Usage Example**

    With auto-detected flow:

    ```bash
    sglang generate \
      --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-w8a8 \
      --prompt "a beautiful sunset" \
      --save-output
    ```

- **Available Quantization Methods**:
    - [x]  ```W4A4_DYNAMIC``` linear with online quantization of activations
    - [x]  ```W8A8``` linear with offline quantization of activations
    - [x]  ```W8A8_DYNAMIC``` linear with online quantization of activations
    - [x]  ```W8A8_MXFP8``` linear with offline quantization (msmodelslim pre-quantized weights)
    - [x]  ```mxfp8``` linear with online quantization (`--quantization mxfp8`)
    - [x]  ```W4A4_MXFP4``` / ```W4A4_MXFP4_DUALSCALE``` linear with offline quantization (msmodelslim pre-quantized weights)
    - [x]  ```mxfp4_npu``` linear with online quantization (`--quantization mxfp4_npu`)

## MXFP8 Online Quantization

For online MXFP8 quantization, load the original FP16/BF16 model and add `--quantization mxfp8`.
Weights are quantized at load time via `npu_dynamic_mx_quant`, and activations are quantized per-token
during inference with `npu_quant_matmul` (block_size=32).

```bash
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --prompt "a fox walking through neon rain" \
  --save-output
```

> **Hardware requirement:** Ascend A5 series or newer. `npu_dynamic_mx_quant` is not available on A2/A3.

## MXFP8 Offline Quantization (msmodelslim)

Pre-quantized MXFP8 weights exported by msmodelslim are auto-detected via `quant_model_description.json`
(`W8A8_MXFP8` scheme). Use `wan_repack.py` to convert the quantized weights to Diffusers format,
then load the converted model with `--model-path`:

```bash
sglang generate \
  --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-mxfp8 \
  --prompt "a beautiful sunset" \
  --save-output
```

## MXFP4 Online Quantization

For online MXFP4 quantization on Ascend NPU, load the original FP16/BF16 model and add
`--quantization mxfp4_npu`. The `mxfp4_npu` key is used for Ascend because `mxfp4`
is reserved for the ROCm/aiter backend.

Weights are quantized at load time via `npu_dynamic_dual_level_mx_quant`, and activations
are quantized per-token during inference before `npu_dual_level_quant_matmul`. MXFP4 uses
dual-level block scales with an L1 block size of 32 and an L0 block size of 512.

```bash
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp4_npu \
  --prompt "a fox walking through neon rain" \
  --save-output
```

> **Hardware requirement:** Ascend A5 series or newer. `npu_dynamic_dual_level_mx_quant`
> and `npu_dual_level_quant_matmul` are not available on A2/A3.
>
> **Note:** Online MXFP4 weight quantization is experimental. The offline msmodelslim
> flow uses pre-quantized weights and may produce different numerical results.

## MXFP4 Offline Quantization (msmodelslim)

Pre-quantized MXFP4 weights exported by msmodelslim are auto-detected via
`quant_model_description.json` (`W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` scheme).
Use `wan_repack.py` to convert the quantized weights to Diffusers format, then load
the converted model with `--model-path`:

```bash
sglang generate \
  --model-path {path_to_converted_mxfp4_model} \
  --prompt "a beautiful sunset" \
  --save-output
```

The offline MXFP4 checkpoint stores weights in an FP8 container and includes dual-level
scales (`weight_scale`, `weight_dual_scale`). If exported with smooth quantization,
`mul_scale` is loaded and applied before activation quantization to keep activations
aligned with the calibrated weights.