[Docs] Update MegaMoE handling and rerun benchmarks (#27726)
This commit is contained in:
zijiexia
@@ -135,7 +135,8 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
|
||||
- `supportedHardware` + the EXAMPLE cells: keep your tested families; **delete the
|
||||
`mi*` ids + AMD example cell if no AMD recipe**, etc. A GPU not in the shared catalog
|
||||
(e.g. RTX PRO 6000) → declare it in `config.hardware` and add its id here.
|
||||
- `playgroundFeatures` axes: delete `megamoe` (non-Blackwell-MoE), `hisparse`
|
||||
- `playgroundFeatures` axes: remove the `megamoe` backend option + the
|
||||
`megamoeQuant` block from the `moe` axis (non-Blackwell-MoE), delete `hisparse`
|
||||
(non-DSA), `pdDisagg`/`router` (no PD), the `parsers` axis (no parsers), etc.
|
||||
- `quantizations` / `variants`: drop what the model doesn't ship; collapse `variants`
|
||||
to single `default` if there's no variant axis (then drop the `variant` half of
|
||||
|
||||
@@ -109,13 +109,12 @@ the `_playground.jsx` header):
|
||||
| Axis key | Widget | Use when |
|
||||
|---|---|---|
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `moe` | Backend select + EP knob | Model is MoE and supports multiple `--moe-*-backend` choices. |
|
||||
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
|
||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||
| `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a separate companion block and retargets the cURL modal to `router.port` (clients hit the router, not the role servers). |
|
||||
| `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. |
|
||||
| `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. |
|
||||
| `megamoe` | Single-select with hw/strategy gating | Blackwell-only kernel fusion variant. |
|
||||
|
||||
**Per-chip constraints**: any chip entry in any axis can be wrapped with
|
||||
`hide` / `disable` constraint objects:
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
# Engine extension: add a new playground feature axis
|
||||
|
||||
Loaded on demand by the `cookbook-add-model` skill. **Rare** — adding a model
|
||||
cookbook is data-only and never needs this. The current 8 built-in axes
|
||||
cookbook is data-only and never needs this. The current 7 built-in axes
|
||||
(`attention`, `moe`, `parsers`, `speculative`, `pdDisagg`, `hicache`,
|
||||
`hisparse`, `megamoe`) already cover the SGLang feature surface most cookbooks
|
||||
need. Only add a new axis if a real cookbook needs it and the feature does not
|
||||
fit any existing axis. Touches `_playground.jsx` only.
|
||||
`hisparse`) already cover the SGLang feature surface most cookbooks need.
|
||||
(MegaMoE is not its own axis — it lives inside `moe` as a backend option +
|
||||
a `megamoeQuant` sub-select.) Only add a new axis if a real cookbook needs it
|
||||
and the feature does not fit any existing axis. Touches `_playground.jsx` only.
|
||||
|
||||
For the per-model config/cells/MDX reference see [authoring-reference.md](authoring-reference.md).
|
||||
|
||||
@@ -94,7 +95,7 @@ Template:
|
||||
// also receives the derived
|
||||
// value (as `derived`) and may use it as a no-op shortcut when the
|
||||
// user's pick matches base. Skip when your axis owns flags that never
|
||||
// appear in base cells (PD-Disagg / HiCache / MegaMoE).
|
||||
// appear in base cells (PD-Disagg / HiCache).
|
||||
// deriveFromBase: (cell, fc, h) => ({ ... }) | null,
|
||||
|
||||
// Optional: hints for the renderer. Currently only pdDisagg uses this
|
||||
@@ -103,7 +104,7 @@ Template:
|
||||
|
||||
// Returns the axis card JSX. The outer div MUST have key={axisId} so
|
||||
// React can track it in the engine's map loop. Return null for
|
||||
// axis-level gating (e.g. MegaMoE on Hopper). Lay out as a single
|
||||
// axis-level gating (e.g. HiSparse when the live PD mode isn't `decode`). Lay out as a single
|
||||
// compact horizontal row: title on the left, fields after.
|
||||
render: ({ axisId, value, setValue, fc, base, s, h, renderChip, renderSelect, derived }) => {
|
||||
if (/* axis-level gating fails */) return null;
|
||||
@@ -147,7 +148,7 @@ Template:
|
||||
the **default** compact control (a `<select>` dropdown). It filters
|
||||
hidden chips and disables greyed-out ones internally — no per-chip
|
||||
`evaluateChip` loop needed in the render body. Most axes use it
|
||||
(attention, moe, pdDisagg, hisparse, hicache, megamoe). Pass
|
||||
(attention, moe, pdDisagg, hisparse, hicache). Pass
|
||||
`{ hideValues: [<sentinel>] }` when your `deriveFromBase` resolved to
|
||||
a real value, so the inherit-sentinel ("Auto" / "Inherited" /
|
||||
"current") doesn't clutter the dropdown.
|
||||
@@ -206,7 +207,8 @@ emits flags that should land somewhere specific, include the most likely
|
||||
anchor prefixes in your call. Order doesn't matter (set semantics).
|
||||
|
||||
**Conditional strips** — Some axes strip ONLY when overridden
|
||||
(`attention.tp`, `moe.backend`, `speculative`, `megamoe`). Others strip
|
||||
(`attention.tp`, `moe.backend` (incl. the MegaMoE quant env), `speculative`).
|
||||
Others strip
|
||||
UNCONDITIONALLY whenever declared (`parsers`, `pdDisagg`, `hicache`). The
|
||||
header comment in `AXIS_HANDLERS` documents which policy each axis uses;
|
||||
follow the same pattern when adding a new axis. If unsure, prefer
|
||||
@@ -214,9 +216,9 @@ conditional strip — it preserves base behavior when the user does not
|
||||
opt in.
|
||||
|
||||
**Closure of `AXIS_HANDLERS`** — Inside a handler method, you can
|
||||
reference `AXIS_HANDLERS.<otherAxis>` for cross-handler calls (megamoe
|
||||
does this for `_gateOpen`). This works because `AXIS_HANDLERS` is in
|
||||
lexical scope. Do NOT use this for general logic — it tightly couples
|
||||
reference `AXIS_HANDLERS.<otherAxis>` for cross-handler calls (no built-in
|
||||
axis currently needs this, but it works because `AXIS_HANDLERS` is in
|
||||
lexical scope). Do NOT use this for general logic — it tightly couples
|
||||
handlers. Reserve it for one handler's helpers shared between its own
|
||||
`render` and `revertHidden`.
|
||||
|
||||
|
||||
@@ -178,10 +178,32 @@ sgl-eval run gsm8k \\
|
||||
options: [
|
||||
{ id: null, label: "Inherited" },
|
||||
{ id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] },
|
||||
// KEEP the MegaMoE option + the megamoeQuant block below ONLY for Blackwell
|
||||
// MoE kernel-fusion models; DELETE both otherwise. requiresHw gates it to
|
||||
// Blackwell (the engine hides it elsewhere); add excludesStrategy: [...] too
|
||||
// for a strategy gate. Selecting MegaMoE reveals the Quantization sub-select.
|
||||
{ id: "megamoe", label: "MegaMoE", flags: ["--moe-a2a-backend megamoe"],
|
||||
requiresHw: ["b200", "b300", "gb200", "gb300"] },
|
||||
{ id: "flashinfer_mxfp4", label: "FlashInfer (MXFP4)", flags: ["--moe-runner-backend flashinfer_mxfp4"] },
|
||||
{ id: "marlin", label: "Marlin (W4A16)", flags: ["--moe-runner-backend marlin"] },
|
||||
],
|
||||
},
|
||||
// MegaMoE quantization sub-select — shown only when backend === "megamoe".
|
||||
// W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env.
|
||||
// DELETE this block if there's no MegaMoE backend option above.
|
||||
megamoeQuant: {
|
||||
stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
|
||||
options: [
|
||||
{ id: "w4a8", label: "W4A8",
|
||||
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
|
||||
{ id: "w4a4", label: "W4A4",
|
||||
env: [
|
||||
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
|
||||
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
|
||||
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
|
||||
] },
|
||||
],
|
||||
},
|
||||
ep: { label: "EP", values: [
|
||||
null, 1, 2, 4, 8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
@@ -275,19 +297,6 @@ sgl-eval run gsm8k \\
|
||||
],
|
||||
defaultHostRatio: 10,
|
||||
},
|
||||
|
||||
// ----- Card: "MegaMoE" ----- KEEP only for Blackwell MoE kernel-fusion models.
|
||||
megamoe: {
|
||||
requiresHw: ["b200", "b300", "gb200", "gb300"],
|
||||
excludesStrategy: ["low-latency", "balanced"],
|
||||
stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
|
||||
options: [
|
||||
{ id: "disabled", label: "Disabled" },
|
||||
{ id: "w4a8", label: "W4A8",
|
||||
flags: ["--moe-a2a-backend megamoe"],
|
||||
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
|
||||
],
|
||||
},
|
||||
},
|
||||
|
||||
// EXAMPLE cells — one per hardware family to show the shape. REPLACE each with
|
||||
|
||||
Reference in New Issue
Block a user