minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[MoE] Add FlashInfer SM90 MXFP4 W4A8 CUTLASS MoE (#34967)

Browse Source 
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
This commit is contained in:
Yuan Luo
2026-08-31 20:04:41 -07:00
committed by GitHub
co-authored by luoyuan.luo
parent 22337e9c56
commit 5b04408784
13 changed files with 662 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+1 -1
View File
@@ -337,7 +337,7 @@ automatically.
Two options are available for running DeepSeek-V4 on Hopper:
- **Original FP4 checkpoints** — apply the W4A16 MoE kernels (Marlin) as the command generator picks for Hopper cells. This path works on both H100 and H200 and is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
- **Original FP4 checkpoints** — run the MoE experts with W4A16 kernels (Marlin or the FlashInfer SM90 CUTLASS runner) as the command generator picks for Hopper cells. With FlashInfer >= 0.6.18 you can instead select the **W4A8** path — MXFP4 weights with FP8 activations via FlashInfer's Humming kernels — by adding `--flashinfer-mxfp4-moe-precision fp8` to `--moe-runner-backend flashinfer_mxfp4`; the low-latency Hopper cells now generate this form. Both work on H100 and H200; FP4 is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
- **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).
On these FP8 checkpoints you can additionally enable the all-FP8 **MegaMoE** path on SM90 for higher long-context / large-decode throughput — see the **SM90 (Hopper) FP8 MegaMoE** note in Configuration Tips below.
docs/docs/advanced_features/server_arguments.mdx
+2 -2
View File
@@ -1808,9 +1808,9 @@ Please consult the documentation below and [server_args.py](https://github.com/s
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--flashinfer-mxfp4-moe-precision`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Choose the computation precision of flashinfer mxfp4 moe</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Choose the computation precision of flashinfer mxfp4 moe. On SM90, `fp8` selects the Humming-style MXFP4-weight x FP8-activation path introduced by FlashInfer #3738 and requires FlashInfer &gt;= 0.6.18.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`default`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>default</code>, <code>bf16</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>default</code>, <code>bf16</code>, <code>fp8</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-flashinfer-allreduce-fusion`</td>
docs/docs/hardware-platforms/ascend-npus/reference/support_features.mdx
+1 -1
View File
@@ -1638,7 +1638,7 @@ click [Server Arguments](../../../advanced_features/server_arguments).
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--flashinfer-mxfp4-moe-precision`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`default`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`default`,<br/> `bf16`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`default`,<br/> `bf16`,<br/> `fp8`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Special for GPU</td>
</tr>
<tr>
docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx
+13 -5
View File
@@ -1904,13 +1904,17 @@ sgl-eval run mmmu_pro \\
{
match: { hw: "h200", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" },
verified: true,
// W4A8 (MXFP4 weights x FP8 activations, FlashInfer Humming kernels);
// requires FlashInfer >= 0.6.18. Falls back: drop the precision flag
// for the W4A16 path, or use --moe-runner-backend marlin.
verificationStatus: "in-progress",
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--moe-runner-backend marlin",
"--moe-runner-backend flashinfer_mxfp4",
"--flashinfer-mxfp4-moe-precision fp8",
"--speculative-algorithm DSPARK",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1918,13 +1922,15 @@ sgl-eval run mmmu_pro \\
},
{
match: { hw: "h200", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
verified: true,
// W4A8 Humming path -- see the flash-official cell above.
verificationStatus: "in-progress",
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--moe-runner-backend marlin",
"--moe-runner-backend flashinfer_mxfp4",
"--flashinfer-mxfp4-moe-precision fp8",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
@@ -1993,13 +1999,15 @@ sgl-eval run mmmu_pro \\
},
{
match: { hw: "h200", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "single" },
verified: true,
// W4A8 Humming path -- see the flash-official cell above.
verificationStatus: "in-progress",
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--moe-runner-backend flashinfer_mxfp4",
"--flashinfer-mxfp4-moe-precision fp8",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",