docs: sync legacy docs/-only updates into docs_new (Mintlify) (#27308)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
46c58b5c70
commit
c6c1f1a29a
@@ -175,6 +175,13 @@ The following table summarizes quantization method support across NVIDIA and AMD
|
||||
<td>Yes</td>
|
||||
<td>Ascend quantization; Uses CANN kernels</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>mxfp8</code> (diffusion)</td>
|
||||
<td>No</td>
|
||||
<td>No</td>
|
||||
<td>Yes (A2/A3)</td>
|
||||
<td>Ascend NPU only; online MXFP8 quantization for diffusion models (e.g., Wan2.2); requires CANN ≥ 8.0.RC3</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -816,6 +823,36 @@ sglang serve --model-path Qwen/Qwen3-30B-A3B \
|
||||
--quantization quark_mxfp4
|
||||
```
|
||||
|
||||
## Diffusion Model Quantization on Ascend NPU
|
||||
|
||||
SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend A5 NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the `sglang serve` / `sglang generate` CLI.
|
||||
|
||||
**Requirements:** Ascend A5, CANN ≥ 8.0.RC3
|
||||
|
||||
### Online MXFP8
|
||||
|
||||
Pass `--quantization mxfp8` to dynamically quantize FP16/BF16 transformer weights to MXFP8 at load time:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--quantization mxfp8 \
|
||||
--num-gpus 4
|
||||
```
|
||||
|
||||
### Offline MXFP8 (ModelSlim)
|
||||
|
||||
Pre-quantize with [msModelSlim](https://gitcode.com/Ascend/msmodelslim) and load the checkpoint directly — the quantization scheme is auto-detected from `quant_model_description.json`:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path /path/to/wan2_2_mxfp8_diffusers \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
For the full quantization + format conversion workflow and a complete list of supported schemes, see [Diffusion Quantization on Ascend NPU](../hardware-platforms/ascend-npus/ascend_npu_quantization#diffusion-model-quantization-on-ascend-npu) and [SGLang-Diffusion Quantization](../sglang-diffusion/quantization#modelslim).
|
||||
|
||||
## Reference
|
||||
|
||||
- [GPTQModel](https://github.com/ModelCloud/GPTQModel)
|
||||
|
||||
Reference in New Issue
Block a user