sglang/docs_new/docs/hardware-platforms/ascend-npus/optimization/quantization.mdx

---
title: "Quantization on Ascend"
metatags:
    description: "Load, export, and serve quantized models on Ascend NPUs with SGLang."
---
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.

SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.

[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Supported</th>
      <th>A3 Supported</th>
      <th>Ascend 950 Products Supported</th>
      <th>Diffusion models</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>W4A4 dynamic</td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
    </tr>
    <tr>
      <td>W8A8 static</td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
    </tr>
    <tr>
      <td>W8A8 dynamic</td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
    </tr>
    <tr>
      <td>MXFP8 (<a href="https://github.com/sgl-project/sglang/pull/20922">Diffusion</a>, <a href="https://github.com/sgl-project/sglang/pull/22352">LLM dense</a>)</td>
      <td>Linear</td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/22338">MXFP4</a></td>
      <td>Linear</td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/23650">MXFP4 W4A8</a></td>
      <td>Linear</td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/23795">MXFP4 W4A4</a></td>
      <td>Linear</td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'blue'}}>WIP</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
    </tr>
    <tr>
      <td>W4A4 dynamic</td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
    </tr>
    <tr>
      <td>W4A8 dynamic</td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
    </tr>
    <tr>
      <td>W8A8 dynamic</td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/20922">MXFP8</a> (LLM MoE)</td>
      <td>MoE</td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'red'}}>x</strong></td>
    </tr>
  </tbody>
</table>

[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Supported</th>
      <th>A3 Supported</th>
      <th>Ascend 950 Products Supported</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>W4A16</td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td>W8A16</td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td>W4A16</td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
  </tbody>
</table>

GPTQ on Ascend support
<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Supported</th>
      <th>A3 Supported</th>
      <th>Ascend 950 Products Supported</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/15203">W4A16</a></td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/15203">W8A16</a></td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/16364">W4A16 MOE</a></td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/16364">W8A16 MOE</a></td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
  </tbody>
</table>

[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)
<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Supported</th>
      <th>A3 Supported</th>
      <th>Ascend 950 Products Supported</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>W4A16</td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td>W8A16</td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td>W4A16</td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td>W8A16</td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
  </tbody>
</table>

Compressed-tensors (LLM Compressor) on Ascend support:
<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Supported</th>
      <th>A3 Supported</th>
      <th>Ascend 950 Products Supported</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/14504">W8A8 dynamic</a></td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/14736">W4A8 dynamic with/without activation clip</a></td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/12759">W4A16 MOE</a></td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td><a href="https://github.com/sgl-project/sglang/pull/14504">W8A8 dynamic</a></td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
  </tbody>
</table>

[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
<table>
  <thead>
    <tr>
      <th>Quantization type</th>
      <th>Layer type</th>
      <th>A2 Supported</th>
      <th>A3 Supported</th>
      <th>Ascend 950 Products Supported</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>All GGUF types (standard, K-quant)</td>
      <td>Linear</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
    <tr>
      <td>All GGUF types (standard, K-quant)</td>
      <td>MoE</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
    </tr>
  </tbody>
</table>

**Usage Examples:**

- Dense model (e.g., Qwen3-14B-Q4_K_M.gguf):

```bash Command
python3 -m sglang.launch_server \
    --model-path Qwen3-14B-Q4_K_M.gguf \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.7 --tp-size 2
```

- MoE model (e.g., Qwen3-30B-A3B-Q4_K_M.gguf):

```bash Command
python3 -m sglang.launch_server \
    --model-path Qwen3-30B-A3B-Q4_K_M.gguf \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 2
```

> **Implementation Notes:**
>
> - GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead).
> - MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation.
> - TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models.

**MXFP8 for LLM dense models (e.g., Qwen3 / Qwen3.5):**

LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend 950 Products or newer (`npu_dynamic_mx_quant` is not available on A2 / A3).

- Online MXFP8 quantization (BF16/FP16 weights → MXFP8 at load time):

```bash Command
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

- Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`):

```bash Command
python3 -m sglang.launch_server \
    --model-path /path/to/Qwen3-8B-W8A8-MXFP8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

> **Implementation Notes:**
> - Online path: `Fp8Config.get_quant_method()` dispatches to `NPUMXFP8LinearMethod`. Weights are quantized once at load via `npu_dynamic_mx_quant(weight, dst_type=torch_npu.float8_e4m3fn)` and pre-transposed to `[in, out]`; activations are per-token quantized at inference and matmul runs via `npu_quant_matmul(..., group_sizes=[1, 1, 32])` (block_size = 32).
> - Offline path: `ModelSlimMXFP8Scheme` loads `float8_e4m3fn` weights + `float8_e8m0fnu` block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (`.data` assignment) — calling `.contiguous()` would physically reorder the pre-quantized layout and break the block-scale mapping.
> - MoE MXFP8 (FusedMoE) for LLMs is documented in **MXFP8 for LLM MoE models** below.

**MXFP8 for LLM MoE models (e.g. Qwen3-30B-A3B / Qwen3.5 MoE):**

LLM MoE W8A8 MXFP8 (FusedMoE) support builds on the dense MXFP8 path. Requires Ascend A5 series or newer — the fused MoE MX kernels (`npu_grouped_matmul_swiglu_quant_v2`, `npu_dynamic_mx_quant`) are A5-only.

- Online MXFP8 quantization (BF16/FP16 expert weights → MXFP8 at load time):

```bash Command
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-30B-A3B \
    --quantization mxfp8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

- Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme). No `--quantization` flag is needed: the `quant_model_description.json` shipped with the checkpoint selects both the ModelSlim path and the scheme automatically.

```bash Command
python3 -m sglang.launch_server \
    --model-path /path/to/Qwen3-30B-A3B-W8A8-MXFP8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

> **Implementation Notes:**
> - Both paths share the per-gmm kernel `NPUMXFP8MoEMethod` (`hardware_backend/npu/quantization/moe_methods.py`), which tells online from offline by weight dtype. Expert weights and their e8m0 block scales are kept as non-contiguous transpose views — calling `.contiguous()` would tank HBM bandwidth.
> - Online path: `Fp8Config.get_quant_method()` dispatches FusedMoE layers to `NPUMXFP8OnlineMoEMethod`, which subclasses `UnquantizedFusedMoEMethod` and overrides only `create_moe_runner` to swap in the MXFP8 kernels — weight creation, weight post-processing and the forward pass are the unquantized Ascend ones. BF16 expert weights `w13`/`w2` are quantized once at load via `npu_dynamic_mx_quant(dst_type=torch.float8_e4m3fn)` (a 3D `[E, N, K]` input is accepted directly).
> - Offline path: `ModelSlimMXFP8MoEScheme` (one instance per weight group) loads `float8_e4m3fn` expert weights + uint8 (e8m0, exponent + 127) block scales. The scale is reshaped `[E, N, K/32] → [E, N, K/64, 2]` (contiguous pairing, matching `npu_dynamic_mx_quant`) then transposed.
> - Forward: `AscendTPDispatcher` runs `npu_moe_init_routing_v2(quant_mode=3)`, which fuses the per-token MX activation quant into routing (e4m3 payload + e8m0 block scale, reshaped to the pair-split layout). `AscendRunnerCore` then runs gmm1 `npu_grouped_matmul_swiglu_quant_v2` (cumulative `group_list`; fuses gate/up + swiglu + requant, so no separate activation step) → gmm2 `npu_grouped_matmul` (count `group_list`). The UE8M0 (`float8_e8m0fnu`) scale dtypes are passed explicitly; the e4m3 `x`/`weight` dtypes are left implicit.
> - **Router gate**: msmodelslim may also quantize `mlp.gate` (`W8A8_MXFP8`). The gate is a `ReplicatedLinear`, so its quantization must be **description-driven**: for the offline `modelslim` path the gate is passed the quant config and dequantized correctly; the online path keeps it in BF16. Loading a quantized gate as BF16 without its block scale scrambles routing and produces garbage output.
> - Where the activation quant happens depends on the dispatcher. On `ascend_tp` it is fused into routing as described above. DeepEP has no MXFP8 dispatch dtype, so it keeps dispatching BF16 and gmm1 quantizes the hidden states itself via `npu_dynamic_mx_quant` before the fused kernel — the two paths reach the same gmm1 input. Only the `ascend_tp` path has been validated end-to-end on Ascend A5.

**MXFP4 W4A8 for LLM dense models (e.g., Qwen3 / Qwen3.5):**

LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend 950 Products or newer.

- Online W4A8 quantization (BF16/FP16 weights → MXFP4 at load time):

```bash Command
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp_w4a8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

- Offline W4A8 quantization (msmodelslim pre-quantized weights, `W4A8_MXFP` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`).

> **Implementation Notes:**
> - Weights are packed FP4 (`float4_e2m1fn_x2`, two nibbles per byte) with a UE8M0 per-block shared exponent (block_size = 32); activations are per-token MXFP8. Matmul runs via `npu_quant_matmul(..., x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[0, 0, 32])`.
> - The packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype` / `input_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject.
> - Online and offline share the same kernel path and layout; they differ only in the weight source (RTN at load vs msmodelslim calibration).

**MXFP4 W4A4 for LLM dense models (e.g. Qwen3 / Qwen3.5):**

LLM dense W4A4 (MXFP4 4-bit weights + 4-bit activations) Linear support was added in [PR #23795](https://github.com/sgl-project/sglang/pull/23795). Requires Ascend A5 series (Ascend 950) or newer — the dual-level online path uses the `DualLevelQuantBatchMatmul` op, which A2/A3 lack. On the Ascend NPU backend `--quantization mxfp4` selects this W4A4 path (on GPU the same flag selects the upstream OCP MXFP4 MoE config instead).

- Online W4A4 quantization (BF16/FP16 weights → dual-level MXFP4 at load time):

```bash Command
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp4 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

- Offline W4A4 quantization (msmodelslim pre-quantized weights, `W4A4_MXFP4` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`).

> **Implementation Notes:**
> - **Online** (`NPUDualLevelMXFP4LinearMethod`) uses **dual-level** MXFP4: both weights and activations are quantized with a fine FP8 (E4M3) L0 block scale plus a coarser L1 scale via `npu_dynamic_dual_level_mx_quant`, and the matmul runs via `npu_dual_level_quant_matmul` (weight in FRACTAL_NZ). Dual-level captures per-block dynamic range far better than a single UE8M0 (power-of-2) scale, which is what made an earlier single-level RTN online path degenerate (greedy decoding could loop without emitting EOS).
> - **Offline** (`ModelSlimMXFP4Scheme` → `NPUSingleLevelMXFP4OfflineLinearMethod`) is **single-level**: msmodelslim's `W4A4_MXFP4` checkpoint ships single-level UE8M0 block scales (block_size = 32), so the matmul runs via `npu_quant_matmul(..., x1_dtype=x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[1, 1, 32])`. The online and offline paths therefore use different matmul kernels — they no longer share the matmul path.
> - As with W4A8, the packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject.
> - Validated end-to-end on Ascend A5 hardware.

## Diffusion Model Quantization on Ascend NPU

SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires Ascend 950 Products; the ModelSlim W8A8/W4A4 schemes work on A2/A3.

**Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend 950 Products

<table>
  <thead>
    <tr>
      <th>Quantization method</th>
      <th><code>quant_type</code> in JSON</th>
      <th>Scheme class</th>
      <th>Mode</th>
      <th>A2/A3 Supported</th>
      <th>Ascend 950 Products Supported</th>
      <th>Trigger</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>MXFP8 (W8A8)</td>
      <td>—</td>
      <td><code>MXFP8Config</code></td>
      <td>Online</td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><code>--quantization mxfp8</code></td>
    </tr>
    <tr>
      <td>MXFP8 (W8A8)</td>
      <td><code>W8A8_MXFP8</code></td>
      <td><code>ModelSlimMXFP8Scheme</code></td>
      <td>Offline</td>
      <td><strong style={{color: 'red'}}>x</strong></td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td>auto-detected from <code>quant_model_description.json</code></td>
    </tr>
    <tr>
      <td>W8A8 static</td>
      <td><code>W8A8</code></td>
      <td><code>ModelSlimW8A8Int8</code></td>
      <td>Offline</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td>auto-detected from <code>quant_model_description.json</code></td>
    </tr>
    <tr>
      <td>W8A8 dynamic</td>
      <td><code>W8A8_DYNAMIC</code></td>
      <td><code>ModelSlimW8A8Int8</code></td>
      <td>Offline</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td>auto-detected from <code>quant_model_description.json</code></td>
    </tr>
    <tr>
      <td>W4A4 dynamic</td>
      <td><code>W4A4_DYNAMIC</code></td>
      <td><code>ModelSlimW4A4Int4</code></td>
      <td>Offline</td>
      <td><strong style={{color: 'green'}}>√</strong></td>
      <td><strong style={{color: 'orange'}}>TBD</strong></td>
      <td>auto-detected from <code>quant_model_description.json</code></td>
    </tr>
  </tbody>
</table>

### Online MXFP8 Quantization

Online quantization dynamically quantizes FP16/BF16 weights to MXFP8 at load time using `npu_dynamic_mx_quant` + `npu_quant_matmul` CANN kernels. Pass `--quantization mxfp8` to override auto-detection.

```bash Command
# Start the diffusion server with online MXFP8 quantization
sglang serve \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --num-gpus 4
```

```bash Command
# One-shot generation
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --prompt "a beautiful sunset over the mountains" \
  --save-output
```

### Offline MXFP8 Quantization (ModelSlim)

For offline quantization, pre-quantize the model with msModelSlim and load the resulting checkpoint. The quantization scheme is auto-detected from `quant_model_description.json`, so no extra `--quantization` flag is needed.

**Step 1: Quantize with msModelSlim**

```bash Command
msmodelslim quant \
  --model_path /path/to/wan2_2_float_weights \
  --save_path /path/to/wan2_2_mxfp8_weights \
  --device npu \
  --model_type Wan2_2 \
  --quant_type mxfp8 \
  --trust_remote_code True
```

> Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim.

**Step 2: Convert to Diffusers format**

msModelSlim saves quantized Wan2.2 weights in the original Wan format. Convert to Diffusers format using the provided repack script:

```bash Command
python python/sglang/multimodal_gen/tools/wan_repack.py \
  --input-path /path/to/wan2_2_mxfp8_weights \
  --output-path /path/to/wan2_2_mxfp8_diffusers
```

Then copy all files from the original Diffusers checkpoint (except the `transformer`/`transformer_2` folders) into the output directory.

**Step 3: Run inference**

```bash Command
sglang generate \
  --model-path /path/to/wan2_2_mxfp8_diffusers \
  --prompt "a beautiful sunset over the mountains" \
  --save-output
```

For pre-quantized checkpoints available on ModelScope, see [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).