minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[Docs] Update MegaMoE handling and rerun benchmarks (#27726)

Browse Source
This commit is contained in:
zijiexia
2026-06-09 19:00:43 -07:00
committed by GitHub
parent 6110ed671f
commit 6565b7c464
8 changed files with 133 additions and 193 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/cookbook-add-model/SKILL.md
+2 -1
View File
@@ -135,7 +135,8 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
- `supportedHardware` + the EXAMPLE cells: keep your tested families; **delete the
`mi*` ids + AMD example cell if no AMD recipe**, etc. A GPU not in the shared catalog
(e.g. RTX PRO 6000) → declare it in `config.hardware` and add its id here.
- `playgroundFeatures` axes: delete `megamoe` (non-Blackwell-MoE), `hisparse`
- `playgroundFeatures` axes: remove the `megamoe` backend option + the
`megamoeQuant` block from the `moe` axis (non-Blackwell-MoE), delete `hisparse`
(non-DSA), `pdDisagg`/`router` (no PD), the `parsers` axis (no parsers), etc.
- `quantizations` / `variants`: drop what the model doesn't ship; collapse `variants`
to single `default` if there's no variant axis (then drop the `variant` half of
.claude/skills/cookbook-add-model/references/authoring-reference.md
+1 -2
View File
@@ -109,13 +109,12 @@ the `_playground.jsx` header):
| Axis key | Widget | Use when |
|---|---|---|
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `moe` | Backend select + EP knob | Model is MoE and supports multiple `--moe-*-backend` choices. |
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
| `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a separate companion block and retargets the cURL modal to `router.port` (clients hit the router, not the role servers). |
| `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. |
| `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. |
| `megamoe` | Single-select with hw/strategy gating | Blackwell-only kernel fusion variant. |
**Per-chip constraints**: any chip entry in any axis can be wrapped with
`hide` / `disable` constraint objects:
.claude/skills/cookbook-add-model/references/engine-axis.md
+13 -11
View File
@@ -1,11 +1,12 @@
# Engine extension: add a new playground feature axis
Loaded on demand by the `cookbook-add-model` skill. **Rare** — adding a model
cookbook is data-only and never needs this. The current 8 built-in axes
cookbook is data-only and never needs this. The current 7 built-in axes
(`attention`, `moe`, `parsers`, `speculative`, `pdDisagg`, `hicache`,
`hisparse`, `megamoe`) already cover the SGLang feature surface most cookbooks
need. Only add a new axis if a real cookbook needs it and the feature does not
fit any existing axis. Touches `_playground.jsx` only.
`hisparse`) already cover the SGLang feature surface most cookbooks need.
(MegaMoE is not its own axis — it lives inside `moe` as a backend option +
a `megamoeQuant` sub-select.) Only add a new axis if a real cookbook needs it
and the feature does not fit any existing axis. Touches `_playground.jsx` only.
For the per-model config/cells/MDX reference see [authoring-reference.md](authoring-reference.md).
@@ -94,7 +95,7 @@ Template:
// also receives the derived
// value (as `derived`) and may use it as a no-op shortcut when the
// user's pick matches base. Skip when your axis owns flags that never
// appear in base cells (PD-Disagg / HiCache / MegaMoE).
// appear in base cells (PD-Disagg / HiCache).
// deriveFromBase: (cell, fc, h) => ({ ... }) | null,
// Optional: hints for the renderer. Currently only pdDisagg uses this
@@ -103,7 +104,7 @@ Template:
// Returns the axis card JSX. The outer div MUST have key={axisId} so
// React can track it in the engine's map loop. Return null for
// axis-level gating (e.g. MegaMoE on Hopper). Lay out as a single
// axis-level gating (e.g. HiSparse when the live PD mode isn't `decode`). Lay out as a single
// compact horizontal row: title on the left, fields after.
render: ({ axisId, value, setValue, fc, base, s, h, renderChip, renderSelect, derived }) => {
if (/* axis-level gating fails */) return null;
@@ -147,7 +148,7 @@ Template:
the **default** compact control (a `<select>` dropdown). It filters
hidden chips and disables greyed-out ones internally — no per-chip
`evaluateChip` loop needed in the render body. Most axes use it
(attention, moe, pdDisagg, hisparse, hicache, megamoe). Pass
(attention, moe, pdDisagg, hisparse, hicache). Pass
`{ hideValues: [<sentinel>] }` when your `deriveFromBase` resolved to
a real value, so the inherit-sentinel ("Auto" / "Inherited" /
"current") doesn't clutter the dropdown.
@@ -206,7 +207,8 @@ emits flags that should land somewhere specific, include the most likely
anchor prefixes in your call. Order doesn't matter (set semantics).
**Conditional strips** — Some axes strip ONLY when overridden
(`attention.tp`, `moe.backend`, `speculative`, `megamoe`). Others strip
(`attention.tp`, `moe.backend` (incl. the MegaMoE quant env), `speculative`).
Others strip
UNCONDITIONALLY whenever declared (`parsers`, `pdDisagg`, `hicache`). The
header comment in `AXIS_HANDLERS` documents which policy each axis uses;
follow the same pattern when adding a new axis. If unsure, prefer
@@ -214,9 +216,9 @@ conditional strip — it preserves base behavior when the user does not
opt in.
**Closure of `AXIS_HANDLERS`** — Inside a handler method, you can
reference `AXIS_HANDLERS.<otherAxis>` for cross-handler calls (megamoe
does this for `_gateOpen`). This works because `AXIS_HANDLERS` is in
lexical scope. Do NOT use this for general logic — it tightly couples
reference `AXIS_HANDLERS.<otherAxis>` for cross-handler calls (no built-in
axis currently needs this, but it works because `AXIS_HANDLERS` is in
lexical scope). Do NOT use this for general logic — it tightly couples
handlers. Reserve it for one handler's helpers shared between its own
`render` and `revertHidden`.
.claude/skills/cookbook-add-model/templates/config.jsx.tmpl
+22 -13
View File
@@ -178,10 +178,32 @@ sgl-eval run gsm8k \\
options: [
{ id: null, label: "Inherited" },
{ id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] },
// KEEP the MegaMoE option + the megamoeQuant block below ONLY for Blackwell
// MoE kernel-fusion models; DELETE both otherwise. requiresHw gates it to
// Blackwell (the engine hides it elsewhere); add excludesStrategy: [...] too
// for a strategy gate. Selecting MegaMoE reveals the Quantization sub-select.
{ id: "megamoe", label: "MegaMoE", flags: ["--moe-a2a-backend megamoe"],
requiresHw: ["b200", "b300", "gb200", "gb300"] },
{ id: "flashinfer_mxfp4", label: "FlashInfer (MXFP4)", flags: ["--moe-runner-backend flashinfer_mxfp4"] },
{ id: "marlin", label: "Marlin (W4A16)", flags: ["--moe-runner-backend marlin"] },
],
},
// MegaMoE quantization sub-select — shown only when backend === "megamoe".
// W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env.
// DELETE this block if there's no MegaMoE backend option above.
megamoeQuant: {
stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
options: [
{ id: "w4a8", label: "W4A8",
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
{ id: "w4a4", label: "W4A4",
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
] },
],
},
ep: { label: "EP", values: [
null, 1, 2, 4, 8,
{ value: 16, disable: { nodes: ["single"] },
@@ -275,19 +297,6 @@ sgl-eval run gsm8k \\
],
defaultHostRatio: 10,
},
// ----- Card: "MegaMoE" ----- KEEP only for Blackwell MoE kernel-fusion models.
megamoe: {
requiresHw: ["b200", "b300", "gb200", "gb300"],
excludesStrategy: ["low-latency", "balanced"],
stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
options: [
{ id: "disabled", label: "Disabled" },
{ id: "w4a8", label: "W4A8",
flags: ["--moe-a2a-backend megamoe"],
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
],
},
},
// EXAMPLE cells — one per hardware family to show the shape. REPLACE each with
.claude/skills/cookbook-review-pr/SKILL.md
+2 -2
View File
@@ -53,8 +53,8 @@ than restating.
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
- `github.cookbookModel` matches the issue-template `model` dropdown value.
- `playgroundFeatures` axes are pruned to what the model supports — no empty/stub axes
(`megamoe` only on Blackwell MoE; `hisparse` only DSA-style; `pdDisagg.router` only with
a PD topology).
(the `moe` axis's MegaMoE backend option + `megamoeQuant` block only on Blackwell MoE,
gated by `requiresHw`; `hisparse` only DSA-style; `pdDisagg.router` only with a PD topology).
- **No leftover `__TOKEN__`** — the config was stamped from the template and every
placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns
nothing).
docs_new/src/snippets/_playground.jsx
+59 -62
View File
@@ -7,13 +7,12 @@
// `config.playgroundFeatures` — a keyed map where each present key opts that
// axis in. Recognised axes:
// attention — TP/CP/DP-Attention knobs
// moe — backend + EP
// moe — backend (+ MegaMoE quantization sub-select) + EP
// parsers — per-item toggle flags
// speculative — single-select preset
// pdDisagg — role + transfer backend + IB device + optional router
// hicache — enable + backend + write policy
// hisparse — enable + host ratio (decode-only)
// megamoe — single-select, Blackwell-only
//
// Adding an axis = one entry in AXIS_HANDLERS below; nothing else switches on
// an axis id. Each handler implements initState / revertHidden / apply /
@@ -354,18 +353,25 @@ export const Playground = ({ config }) => {
},
// ---- Axis: MoE Parallelism ----------------------------------------------
// Backend single-select + EP numeric knob; either is optional.
// Backend single-select + EP numeric knob; either is optional. Picking the
// "megamoe" backend reveals a Quantization sub-select (W4A8 / W4A4) in the same
// row — W4A4 adds the FP4-activations env vars.
moe: {
initState: () => ({ backend: null, ep: null }),
initState: () => ({ backend: null, ep: null, mmQuant: null }),
// Prefer --moe-a2a-backend over --moe-runner-backend when both present.
// mmQuant is derived from the base env (FP4 activations present → W4A4).
deriveFromBase: (cell, fc, h) => {
const flags = (cell && cell.flags) || [];
const baseEnv = (cell && cell.env) || [];
const a2a = h.findFlagArg(flags, "--moe-a2a-backend");
const runner = h.findFlagArg(flags, "--moe-runner-backend");
const fp4Acts = baseEnv.some(
(e) => e.startsWith("SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS"));
return {
backend: a2a || runner || null,
ep: h.parseIntFlag(flags, "--ep"),
mmQuant: fp4Acts ? "w4a4" : "w4a8",
};
},
@@ -376,6 +382,15 @@ export const Playground = ({ config }) => {
&& h.isHidden(fc.backend.options, next.backend, base)) {
next.backend = null; changed = true;
}
// MegaMoE backend availability — gated by its option's requiresHw /
// excludesStrategy (this model gates by hw only; the check is generic).
const mmOpt = (fc.backend?.options || []).find((o) => o.id === "megamoe");
const mmAvail = !!mmOpt
&& (!mmOpt.requiresHw || mmOpt.requiresHw.includes(base.hw))
&& (!mmOpt.excludesStrategy || !mmOpt.excludesStrategy.includes(base.strategy));
if (next.backend === "megamoe" && !mmAvail) {
next.backend = null; changed = true;
}
if (next.ep !== null && fc.ep?.values
&& h.isHidden(fc.ep.values, next.ep, base)) {
next.ep = null; changed = true;
@@ -383,7 +398,7 @@ export const Playground = ({ config }) => {
return changed ? next : value;
},
apply: ({ flags, env, value, fc, h }) => {
apply: ({ flags, env, value, fc, h, derived }) => {
if (value.backend !== null) {
flags = h.stripFlagsByFirstToken(flags, [
"--moe-a2a-backend", "--moe-runner-backend",
@@ -393,6 +408,28 @@ export const Playground = ({ config }) => {
flags = h.insertAfter(flags, h.ANCHOR_NEAR_DPATTN, opt.flags);
}
}
// MegaMoE owns the MoE path: when the effective backend is megamoe, strip the
// DeepEP dispatch + any prior megamoe env, then re-add the selected quant's
// env. When the backend is explicitly switched away from megamoe, only drop
// the megamoe quant env (leave DeepEP dispatch intact).
const mq = fc.megamoeQuant;
if (mq) {
const quantKeys = [];
for (const o of (mq.options || [])) {
for (const e of (o.env || [])) quantKeys.push(e.split("=")[0]);
}
const effBackend = value.backend !== null
? value.backend : (derived && derived.backend);
if (effBackend === "megamoe") {
env = h.stripEnvByPrefix(env, [...(mq.stripEnv || []), ...quantKeys]);
const quant = value.mmQuant != null
? value.mmQuant : ((derived && derived.mmQuant) || "w4a8");
const opt = (mq.options || []).find((o) => o.id === quant);
if (opt?.env?.length) env = [...env, ...opt.env];
} else if (value.backend !== null) {
env = h.stripEnvByPrefix(env, quantKeys);
}
}
if (value.ep !== null) {
flags = h.stripFlagsByFirstToken(flags, ["--ep"]);
if (value.ep > 1) {
@@ -416,6 +453,12 @@ export const Playground = ({ config }) => {
const d = derived ? derived[k] : null;
return (d !== null && d !== undefined) ? [null] : [];
};
// Hide the MegaMoE backend option where its requiresHw / excludesStrategy exclude this base.
const mmOpt = (fc.backend?.options || []).find((o) => o.id === "megamoe");
const mmAvail = !!mmOpt
&& (!mmOpt.requiresHw || mmOpt.requiresHw.includes(base.hw))
&& (!mmOpt.excludesStrategy || !mmOpt.excludesStrategy.includes(base.strategy));
const backendIsMega = slotDisplay("backend") === "megamoe";
return (
<div key={axisId} style={s.card}>
<div style={s.compactRow}>
@@ -425,7 +468,16 @@ export const Playground = ({ config }) => {
<span style={s.fieldLabel}>Backend</span>
{renderSelect(slotDisplay("backend"), fc.backend.options || [],
(v) => setSlot("backend", v), base, undefined,
{ hideValues: hideNull("backend") })}
{ hideValues: [...hideNull("backend"), ...(mmAvail ? [] : ["megamoe"])] })}
</span>
)}
{fc.megamoeQuant && backendIsMega && (
<span style={s.field}>
<span style={s.fieldLabel}>Quantization</span>
{renderSelect(
value.mmQuant != null ? value.mmQuant : ((derived && derived.mmQuant) || "w4a8"),
fc.megamoeQuant.options || [],
(v) => setSlot("mmQuant", v), base)}
</span>
)}
{fc.ep && (
@@ -863,60 +915,6 @@ export const Playground = ({ config }) => {
},
},
// ---- Axis: MegaMoE ------------------------------------------------------
// Single-select with axis-level gating (requiresHw / excludesStrategy)
// plus per-option hide constraints and env mutation (stripEnv + option.env).
megamoe: {
initState: () => "disabled",
revertHidden: (value, fc, base, h) => {
const hwGate = !fc.requiresHw || fc.requiresHw.includes(base.hw);
const stratGate = !fc.excludesStrategy || !fc.excludesStrategy.includes(base.strategy);
if (!hwGate || !stratGate) {
return value === "disabled" ? value : "disabled";
}
if (value !== "disabled" && h.isHidden(fc.options || [], value, base)) {
return "disabled";
}
return value;
},
apply: ({ flags, env, value, fc, h }) => {
if (!value || value === "disabled") return { flags, env };
const opt = (fc.options || []).find((o) => o.id === value);
if (!opt) return { flags, env };
flags = h.stripFlagsByFirstToken(flags, [
"--moe-a2a-backend", "--moe-runner-backend",
]);
if (opt.flags?.length) {
flags = h.insertAfter(flags, h.ANCHOR_NEAR_DPATTN, opt.flags);
}
const ownedEnvKeys = [...(fc.stripEnv || [])];
for (const o of (fc.options || [])) {
for (const e of (o.env || [])) ownedEnvKeys.push(e.split("=")[0]);
}
env = h.stripEnvByPrefix(env, ownedEnvKeys);
if (opt.env?.length) env = [...env, ...opt.env];
return { flags, env };
},
render: ({ axisId, value, setValue, fc, base, s, renderSelect }) => {
const hwGate = !fc.requiresHw || fc.requiresHw.includes(base.hw);
const stratGate = !fc.excludesStrategy || !fc.excludesStrategy.includes(base.strategy);
if (!hwGate || !stratGate) return null;
return (
<div key={axisId} style={s.card}>
<div style={s.compactRow}>
<span style={s.axisTitle}>MegaMoE</span>
<span style={s.field}>
{renderSelect(value, fc.options || [], setValue, base)}
</span>
</div>
</div>
);
},
},
};
// ==========================================================================
@@ -1527,8 +1525,7 @@ export const Playground = ({ config }) => {
const dpAttnOn = (effDpAttn === true)
|| (typeof effDpAttn === "number" && effDpAttn > 0);
const pdMode = (deltas.pdDisagg && deltas.pdDisagg.mode) || "off";
const megamoeOn = !!(deltas.megamoe && deltas.megamoe !== "disabled");
const constraintBase = { ...base, dpAttnOn, pdMode, megamoeOn };
const constraintBase = { ...base, dpAttnOn, pdMode };
let baseCommand = "";
let playgroundCommand = "";
docs_new/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx
+12 -54
View File
@@ -10,9 +10,9 @@ export const benchmarks = [
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
ttft_ms: 428, tpot_ms: 3.53, tokens_per_sec_per_gpu: 44 },
ttft_ms: 87, tpot_ms: 3.68, tokens_per_sec_per_gpu: 65 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 3111, tpot_ms: 23.82, tokens_per_sec_per_gpu: 121 },
ttft_ms: 290, tpot_ms: 6.21, tokens_per_sec_per_gpu: 489 },
],
},
{
@@ -30,9 +30,9 @@ export const benchmarks = [
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
ttft_ms: 105918, tpot_ms: 70.73, tokens_per_sec_per_gpu: 881 },
ttft_ms: 99949, tpot_ms: 67.46, tokens_per_sec_per_gpu: 939 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
ttft_ms: 273356, tpot_ms: 71.61, tokens_per_sec_per_gpu: 889 },
ttft_ms: 253310, tpot_ms: 66.11, tokens_per_sec_per_gpu: 964 },
],
},
{
@@ -59,9 +59,9 @@ export const benchmarks = [
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
ttft_ms: 205, tpot_ms: 3.43, tokens_per_sec_per_gpu: 54 },
ttft_ms: 88, tpot_ms: 3.67, tokens_per_sec_per_gpu: 66 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 1856, tpot_ms: 14.82, tokens_per_sec_per_gpu: 205 },
ttft_ms: 266, tpot_ms: 6.06, tokens_per_sec_per_gpu: 495 },
],
},
{
@@ -79,9 +79,9 @@ export const benchmarks = [
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
ttft_ms: 82556, tpot_ms: 55.37, tokens_per_sec_per_gpu: 1130 },
ttft_ms: 97028, tpot_ms: 65.09, tokens_per_sec_per_gpu: 966 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
ttft_ms: 207987, tpot_ms: 54.05, tokens_per_sec_per_gpu: 1171 },
ttft_ms: 243335, tpot_ms: 63.98, tokens_per_sec_per_gpu: 998 },
],
},
{
@@ -89,9 +89,9 @@ export const benchmarks = [
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
ttft_ms: 239, tpot_ms: 5.04, tokens_per_sec_per_gpu: 24 },
ttft_ms: 261, tpot_ms: 5.01, tokens_per_sec_per_gpu: 23 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 830, tpot_ms: 15.55, tokens_per_sec_per_gpu: 101 },
ttft_ms: 364, tpot_ms: 11.37, tokens_per_sec_per_gpu: 137 },
],
},
{
@@ -106,26 +106,12 @@ export const benchmarks = [
},
{
match: { hw: "b300", variant: "pro", quant: "fp4", strategy: "high-throughput", nodes: "single" },
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
ttft_ms: 99139, tpot_ms: 44.37, tokens_per_sec_per_gpu: 476 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
ttft_ms: 241544, tpot_ms: 43.51, tokens_per_sec_per_gpu: 492 },
],
},
// ====================================================================
// GB200 + FP4
// ====================================================================
{
match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
ttft_ms: 335, tpot_ms: 3.67, tokens_per_sec_per_gpu: 47 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 2440, tpot_ms: 15.95, tokens_per_sec_per_gpu: 163 },
],
},
{
match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" },
@@ -139,23 +125,9 @@ export const benchmarks = [
},
{
match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" },
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
ttft_ms: 128397, tpot_ms: 84.95, tokens_per_sec_per_gpu: 757 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
ttft_ms: 330479, tpot_ms: 86.7, tokens_per_sec_per_gpu: 741 },
],
},
{
match: { hw: "gb200", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "multi-2" },
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
ttft_ms: 343, tpot_ms: 6.47, tokens_per_sec_per_gpu: 18 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 1345, tpot_ms: 23.85, tokens_per_sec_per_gpu: 65 },
],
},
{
match: { hw: "gb200", variant: "pro", quant: "fp4", strategy: "balanced", nodes: "multi-2" },
@@ -168,13 +140,6 @@ export const benchmarks = [
// ====================================================================
{
match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
ttft_ms: 380, tpot_ms: 4.4, tokens_per_sec_per_gpu: 38 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 2960, tpot_ms: 21.26, tokens_per_sec_per_gpu: 125 },
],
},
{
match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" },
@@ -191,20 +156,13 @@ export const benchmarks = [
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
ttft_ms: 146954, tpot_ms: 97.24, tokens_per_sec_per_gpu: 662 },
ttft_ms: 154868, tpot_ms: 104.84, tokens_per_sec_per_gpu: 621 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
ttft_ms: 368557, tpot_ms: 99.33, tokens_per_sec_per_gpu: 651 },
ttft_ms: 386489, tpot_ms: 103.37, tokens_per_sec_per_gpu: 627 },
],
},
{
match: { hw: "gb300", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "single" },
sglang_version: "0.5.12.post1",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
ttft_ms: 363, tpot_ms: 6.53, tokens_per_sec_per_gpu: 36 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 1275, tpot_ms: 20.75, tokens_per_sec_per_gpu: 152 },
],
},
{
match: { hw: "gb300", variant: "pro", quant: "fp4", strategy: "balanced", nodes: "single" },
docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx
+22 -48
View File
@@ -68,7 +68,8 @@ export const config = {
--model {{MODEL_NAME}} \\
--dataset-name {{DATASET}} \\
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}}`,
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\
--warmup-requests 64`,
accuracy: {
gsm8k_pct:
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
@@ -112,7 +113,7 @@ sgl-eval run aime25 \\
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`,
},
},
numPromptsByConc: { 1: 8, 16: 32, 64: 128, 256: 512, 1024: 2048, 4096: 4096 },
numPromptsByConc: { 1: 32, 16: 32, 64: 128, 256: 512, 1024: 2048, 4096: 4096 },
},
// Per-variant accuracy applied to every cell; per-cell `accuracy` overrides.
@@ -182,21 +183,29 @@ sgl-eval run aime25 \\
options: [
{ id: null, label: "Inherited" },
{ id: "deepep", label: "DeepEP",
flags: ["--moe-a2a-backend deepep"],
disable: { megamoeOn: [true] },
disableReason: "MegaMoE owns the MoE backend — turn MegaMoE off to pick one." },
flags: ["--moe-a2a-backend deepep"] },
// Blackwell-only; no strategy gate — the Playground allows MegaMoE on any
// strategy for experimentation (docs recommend it on high-throughput).
{ id: "megamoe", label: "MegaMoE",
flags: ["--moe-a2a-backend megamoe"],
disable: { megamoeOn: [true] },
disableReason: "MegaMoE owns the MoE backend — turn MegaMoE off to pick one." },
requiresHw: ["b200", "b300", "gb200", "gb300"] },
{ id: "flashinfer_mxfp4", label: "FlashInfer (MXFP4)",
flags: ["--moe-runner-backend flashinfer_mxfp4"],
disable: { megamoeOn: [true] },
disableReason: "MegaMoE owns the MoE backend — turn MegaMoE off to pick one." },
flags: ["--moe-runner-backend flashinfer_mxfp4"] },
{ id: "marlin", label: "Marlin (W4A16)",
flags: ["--moe-runner-backend marlin"],
disable: { megamoeOn: [true] },
disableReason: "MegaMoE owns the MoE backend — turn MegaMoE off to pick one." },
flags: ["--moe-runner-backend marlin"] },
],
},
megamoeQuant: {
stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
options: [
{ id: "w4a8", label: "W4A8",
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
{ id: "w4a4", label: "W4A4",
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
] },
],
},
ep: { label: "EP", values: [
@@ -305,27 +314,6 @@ sgl-eval run aime25 \\
],
defaultHostRatio: 10,
},
// ----- Card 8: "MegaMoE" -----
// Blackwell-only, high-throughput only (see requiresHw / excludesStrategy).
megamoe: {
requiresHw: ["b200", "b300", "gb200", "gb300"],
excludesStrategy: ["low-latency", "balanced"],
stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
options: [
{ id: "disabled", label: "Disabled" },
{ id: "w4a8", label: "W4A8",
flags: ["--moe-a2a-backend megamoe"],
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
{ id: "w4a4", label: "W4A4",
flags: ["--moe-a2a-backend megamoe"],
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
] },
],
},
},
cells: [
@@ -377,8 +365,6 @@ sgl-eval run aime25 \\
verified: true,
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
],
flags: [
"--trust-remote-code",
@@ -442,8 +428,6 @@ sgl-eval run aime25 \\
verified: true,
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
],
flags: [
"--trust-remote-code",
@@ -508,8 +492,6 @@ sgl-eval run aime25 \\
verified: true,
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
],
flags: [
"--trust-remote-code",
@@ -573,8 +555,6 @@ sgl-eval run aime25 \\
verified: true,
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
],
flags: [
"--trust-remote-code",
@@ -642,8 +622,6 @@ sgl-eval run aime25 \\
verified: true,
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
],
flags: [
"--trust-remote-code",
@@ -778,8 +756,6 @@ sgl-eval run aime25 \\
verified: true,
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
],
flags: [
"--trust-remote-code",
@@ -841,8 +817,6 @@ sgl-eval run aime25 \\
verified: true,
env: [
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
],
flags: [
"--trust-remote-code",