[XPU] Support GPT-OSS MXFP4 checkpoints on Intel XPU (#35751)
Co-authored-by: Meng, Hengyu <hengyu.meng@intel.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Meng, Hengyu
Claude Opus 5
parent
d122ca99b2
commit
72078cd7f5
@@ -47,7 +47,7 @@ The following table summarizes quantization method support across NVIDIA and AMD
|
||||
<td>Yes</td>
|
||||
<td>Yes</td>
|
||||
<td>Yes (A5)</td>
|
||||
<td>On GPU: requires CDNA3/CDNA4 with MXFP support (uses Aiter). On Ascend NPU (A5): W4A4 MXFP4 for Qwen3 dense and MoE LLMs (MXFP4 weights + activations) — dense models support online dual-level MXFP4; offline <code>W4A4_MXFP4</code> dense and MoE checkpoints (single-level) are auto-detected via <code>modelslim</code></td>
|
||||
<td>On GPU: requires CDNA3/CDNA4 with MXFP support (uses Aiter). On Ascend NPU (A5): W4A4 MXFP4 for Qwen3 dense and MoE LLMs (MXFP4 weights + activations) — dense models support online dual-level MXFP4; offline <code>W4A4_MXFP4</code> dense and MoE checkpoints (single-level) are auto-detected via <code>modelslim</code>. On Intel GPUs (XPU): W4A16 MoE experts on Xe2/BMG via <code>sgl-kernel-xpu</code>, enabled automatically with <code>--device xpu</code> (see <a href="../hardware-platforms/xpu">Intel GPUs (XPU)</a>)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>mxfp8</code></td>
|
||||
|
||||
@@ -41,6 +41,9 @@ A list of LLMs have been optimized on Intel GPU, and more are on the way:
|
||||
**Note:** The model identifiers listed in the table above
|
||||
have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html).
|
||||
|
||||
Quantized MoE models are covered separately in
|
||||
[MXFP4 MoE Quantization](#mxfp4-moe-quantization) below.
|
||||
|
||||
## Installation
|
||||
|
||||
### Install From Source
|
||||
@@ -116,6 +119,40 @@ sglang serve \
|
||||
--page-size \ # intel_xpu attention backend supports [32, 64, 128]
|
||||
```
|
||||
|
||||
## MXFP4 MoE Quantization
|
||||
|
||||
Native MXFP4 MoE checkpoints (OCP microscaling FP4: packed `e2m1` weights plus
|
||||
per-32-element `ue8m0` block scales) run on Intel GPUs through the
|
||||
`sgl-kernel-xpu` W4A16 grouped GEMM. The expert weights stay in the checkpoint's
|
||||
packed layout end to end — there is no dequantization to BF16 — so GPT-OSS-20B
|
||||
loads in roughly 13 GB rather than the ~42 GB a BF16 upcast would need, which is
|
||||
what lets it fit on a single 24 GB card.
|
||||
|
||||
The `mxfp4` method is registered automatically on `--device xpu`, which already
|
||||
requires `sgl-kernel-xpu` — no extra flag is needed:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path openai/gpt-oss-20b \
|
||||
--device xpu \
|
||||
--attention-backend intel_xpu
|
||||
```
|
||||
|
||||
**Tested models:**
|
||||
|
||||
| Model | Notes |
|
||||
|:---|:---|
|
||||
| [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | 32 experts, top-k 4, hidden 2880, intermediate 2880; clamped-swiglu activation (`gemm1_alpha` 1.702, limit 7.0) with expert biases |
|
||||
|
||||
**Requirements and limitations:**
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| GPU architecture | Xe2 / BMG only (Intel® Arc™ B-Series and Arc™ Pro B-Series) |
|
||||
| Quantized layers | Fused MoE experts only; attention, router, embeddings and LM head stay BF16 (per the checkpoint's `modules_to_not_convert`) |
|
||||
| Activations | BF16 (W4A16). Clamped swiglu (GPT-OSS) and plain SiLU are supported |
|
||||
| Expert parallelism | Not yet validated on this path |
|
||||
|
||||
## Benchmarking with Requests
|
||||
|
||||
You can benchmark the performance via the `bench_serving` script.
|
||||
|
||||
Reference in New Issue
Block a user