minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Support online MXFP8 quantization for ungated MoE (#27939)

Browse Source 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
This commit is contained in:
Brayden Zhong
2026-06-24 16:58:48 -07:00
committed by GitHub
co-authored by Brayden Zhong
parent e4bf0043fe
commit f82addd4a8
4 changed files with 27 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/advanced_features/quantization.mdx
+3 -1
View File
@@ -200,7 +200,7 @@ On Ascend, various layers quantization configurations are supported, see [Ascend
## GEMM Backends for FP4/FP8 Quantization
<Note>
Backend selection is supported only for **blockwise FP8** and **NVFP4** GEMM. When running FP8 or FP4 quantized models, you can select the GEMM backend via `--fp8-gemm-backend` and `--fp4-gemm-backend`.
Backend selection applies to **blockwise FP8**, **MXFP8** (dense linear), and **NVFP4** GEMM. When running offline or online FP8 or FP4 quantized models, you can select the GEMM backend via `--fp8-gemm-backend` and `--fp4-gemm-backend`.
</Note>
### `--fp8-gemm-backend` (Blockwise FP8 GEMM)
@@ -259,6 +259,8 @@ Backend selection is supported only for **blockwise FP8** and **NVFP4** GEMM. Wh
**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM90/SM100/120); 4) AITER (AMD); 5) Triton. **Exception:** SM120 always resolves to Triton.
**MXFP8 dense linear:** `auto` uses `flashinfer_cutlass` on SM100 (else `triton`). `flashinfer_cutlass` is fastest on most shapes; `flashinfer_trtllm` is faster only at small M.
### `--fp4-gemm-backend` (NVFP4 GEMM)
<table>