minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

Implement SM120 DeepSeek V4 flashinfer_mxfp4 moe runner backend + TP2 (#30272)

Browse Source
This commit is contained in:
Gabriel Wu
2026-07-18 03:01:06 -07:00
committed by GitHub
parent 6c6175fabd
commit faf6894093
18 changed files with 502 additions and 233 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+3 -4
View File
@@ -149,7 +149,7 @@ import { Playground } from "/src/snippets/_playground.jsx";
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash">DeepSeek-V4-Flash</a></strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>284B</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); H100 (TP=8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); RTX PRO 6000 (TP=2); H100 (TP=8)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro">DeepSeek-V4-Pro</a></strong></td>
@@ -293,9 +293,8 @@ TCP, which can lead to garbled KV transfer on large checkpoints.
**RTX PRO 6000 (SM120 / Blackwell Desktop) note**
RTX PRO 6000 (96 GB) runs **Flash only** — V4-Pro doesn't fit on 8× 96 GB. It uses the
**low-latency / TP-only** recipe (TP=4, single node) with the **Marlin** W4A16 MoE runner and
`--mem-fraction-static 0.70`; the Deploy panel greys out the other recipes for this card.
RTX PRO 6000 (96 GB) runs **Flash only** with the FlashInfer MXFP4 MoE runner.
V4-Pro doesn't fit on 8× 96 GB; the Deploy panel greys out unsupported recipes.
HiCache and MegaMoE are **not** supported on RTX PRO 6000.
**AMD (MI300X / MI355X) note**
docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx
+3 -4
View File
@@ -1421,7 +1421,6 @@ sgl-eval run aime25 \\
// ====================================================================
// RTX PRO 6000 (SM120 / Blackwell Desktop) — Flash + low-latency only
// (V4-Pro doesn't fit on 8× 96 GB); TP-only, Marlin MoE runner.
// ====================================================================
{
match: { hw: "rtx6000", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
@@ -1430,9 +1429,9 @@ sgl-eval run aime25 \\
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--moe-runner-backend marlin",
"--mem-fraction-static 0.70",
"--tp 2",
"--moe-runner-backend flashinfer_mxfp4",
"--mem-fraction-static 0.92",
"--cuda-graph-max-bs-decode 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",