From 33f99831f8723098296c3028890db46c9c1a3082 Mon Sep 17 00:00:00 2001 From: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com> Date: Mon, 15 Jun 2026 08:15:38 +0100 Subject: [PATCH] docs(minimax-m3): refresh B200 benchmarks (tp8, piecewise) + add GPQA (#28207) Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com> --- .../autoregressive/MiniMax/MiniMax-M3.mdx | 18 +++---- .../MiniMaxAI/minimax-m3-benchmarks.jsx | 47 ++++++++++--------- .../snippets/configs/MiniMaxAI/minimax-m3.jsx | 22 ++++++--- 3 files changed, 52 insertions(+), 35 deletions(-) diff --git a/docs_new/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx b/docs_new/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx index ce83e1216..7fe23c1d3 100644 --- a/docs_new/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx +++ b/docs_new/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx @@ -42,7 +42,7 @@ docker pull lmsysorg/sglang:dev-cu13-minimax-m3 The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `…-rocm700-mi30x`, MI350X/MI355X → `…-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces. -These images do not yet bundle MiniMax's MSA sparse-attention kernel (bundling it by default is in progress). Blackwell users who want the recommended fast path install MSA manually after pulling — see **§2.1**. Without it, the same recipe still serves on the built-in Triton sparse path. +These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path. @@ -79,7 +79,7 @@ Key characteristics as served by SGLang: - **Sparse attention**: most layers use M3's "lightning indexer" block-sparse attention (top-k 128-token blocks), which keeps decode cost roughly flat in context length. On Blackwell, MiniMax's open-source [MSA kernel](https://github.com/MiniMax-AI/MSA) accelerates this path further (§2.1). - **MXFP8 quantization across vendors**: the MXFP8 MoE weights run natively on NVIDIA Blackwell (B200 / B300 / GB200 / GB300) and on AMD Instinct MI350X/MI355X (gfx950 / CDNA4), both of which have hardware MX-scaled matmul. On AMD MI300X/MI325X (gfx942 / CDNA3) — no hardware MX — SGLang converts the weights to block-fp8 `[128,128]` at load and serves them on the tuned ROCm kernels (§2.3). The vision tower stays unquantized. -**Recommended generation** (from the model card): `temperature` 1.0, `top_p` 0.95, `top_k` 40 — SGLang applies these automatically from the model's `generation_config.json`. +**Recommended generation**: the model's `generation_config.json` sets `temperature` 1.0 / `top_p` 0.95, which SGLang applies automatically (the default `--sampling-defaults model`). The model card additionally suggests `top_k` 40, but that value is **not** in `generation_config.json`, so SGLang does not apply it by default. `top_k` is a per-request sampling parameter (not a launch flag) — set it per call if you want it, e.g. `extra_body={"top_k": 40}` with the OpenAI client. **Resources:** [HuggingFace](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) · [MSA kernel](https://github.com/MiniMax-AI/MSA) @@ -87,7 +87,7 @@ Key characteristics as served by SGLang: ### 2.1 MSA sparse-attention fast path (recommended for Blackwell users) -[MiniMax MSA](https://github.com/MiniMax-AI/MSA) (`fmha_sm100`, MIT-licensed) is the recommended Blackwell kernel for M3's main sparse-attention step — faster and more memory-efficient than the built-in Triton fallback. It is purely additive — install it and the recipe above engages it automatically; without it the same recipe still serves on the built-in Triton path. The swap is numerically equivalent (cosine ≥ 0.99999 vs Triton), decode stays CUDA-graph-capturable, prefill TTFT drops ~9–12% at 8K–64K context, and the MSA path survives memory configurations where the Triton path OOMs. +[MiniMax MSA](https://github.com/MiniMax-AI/MSA) (`fmha_sm100`, MIT-licensed) is the recommended Blackwell kernel for M3's main sparse-attention step — faster and more memory-efficient than the built-in Triton fallback. **It ships pre-installed in the M3 dev image** (`lmsysorg/sglang:dev-minimax-m3`, also published under the `dev-cu13-minimax-m3` tag), so the Blackwell recipe above engages it automatically with no extra setup — `import fmha_sm100` works out of the box and the kernels JIT-compile on first use. It is otherwise purely additive: on a custom image, install it (below) and the recipe engages it automatically; without it the same recipe still serves on the built-in Triton path. The swap is numerically equivalent (cosine ≥ 0.99999 vs Triton), decode stays CUDA-graph-capturable, prefill TTFT drops ~9–12% at 8K–64K context, and the MSA path survives memory configurations where the Triton path OOMs. **Requirements** (from the [MSA README](https://github.com/MiniMax-AI/MSA#requirements)): @@ -95,10 +95,12 @@ Key characteristics as served by SGLang: - **Toolchain**: CUDA Toolkit with `nvcc` ≥ 12.x on `PATH` (or `CUDA_HOME` set) — the kernels are JIT-compiled at first import. - **Python**: ≥ 3.10; **OS**: Linux — works on both **x86_64 and aarch64 (Grace, e.g. GB200 / GB300)**; the aarch64 build needs no source edits. - + + +The M3 Blackwell dev images above already bundle MSA, so you can skip straight to the gate check. The `git clone` / `pip install` steps are only needed on a custom image that doesn't have `fmha_sm100`. ```bash Command -# --recursive pulls the CUTLASS submodule required for JIT compilation +# Only on a custom image: --recursive pulls the CUTLASS submodule required for JIT compilation git clone --recursive https://github.com/MiniMax-AI/MSA.git msa cd msa && pip install . # Verify the SGLang gate (True -> MSA engaged on this device; False -> Triton fallback): @@ -123,11 +125,11 @@ For multimodal (image) serving, keep the same text recipe above — `--attention ### 2.2 Memory and workload tuning -The NVIDIA Blackwell recipe is the validated single-node **4-GPU (`--tp 4`)** config, which is also the GB200 / GB300 single-node ceiling. It runs identically on B200 (sm_100), B300 (sm_103), and GB300 (sm_103, aarch64); GB200 (sm_100, aarch64) is inferred-supported — both of its axes are validated above — but not directly benchmarked. The AMD recipes use **8-GPU (`--tp 8`)**. +The NVIDIA Blackwell recipes are validated single-node: **B200 at `--tp 8`** and **B300 / GB300 at `--tp 4`** (4-GPU is also the GB200 / GB300 single-node ceiling). GB200 (sm_100, aarch64) is inferred-supported — both of its axes are validated above (B200 is sm_100; GB300 is sm_103 aarch64) — but not directly benchmarked. The AMD recipes use **8-GPU (`--tp 8`)**. -- **Memory**: `--mem-fraction-static` trades KV-pool capacity against prefill **activation headroom** — `0.75` is the safe default on NVIDIA (`0.80` on AMD). A higher value is fine at low concurrency but OOMs under high concurrency or long context, so raise it only for interactive single-stream serving. +- **Memory**: `--mem-fraction-static` reserves GPU memory for weights + KV pool; the rest is prefill **activation headroom**. The value scales with *free* memory per GPU (card capacity minus per-GPU weight), so it tracks the card more than the TP degree: **`0.65` on B200** (180 GB — less headroom once weights are resident) and **`0.75` on the larger-memory B300 / GB300** (`0.80` on AMD). Lower TP packs more weight per GPU, so a tighter config needs a *lower* value — B200 needs `0.65` even at `--tp 4`. Raising it past the validated value is fine only for low-concurrency single-stream serving; it OOMs under high concurrency or long context. - **Long context (32K+)**: keep `--mem-fraction-static` at the platform default and raise `--chunked-prefill-size` to `16384`. Decode TPOT stays roughly flat in context length thanks to sparse attention; 1K–128K prompts are validated. -- **8-GPU nodes**: B200 / B300 hosts with 8 GPUs can use `--tp 8` for more throughput / KV headroom; tp4 is documented as the NVIDIA cross-family common denominator. +- **Scaling TP**: B200 is documented at `--tp 8`; B300 / GB200 / GB300 at `--tp 4` (the single-node cross-family common denominator). On an 8-GPU B300 host you can also raise to `--tp 8` for more throughput / KV headroom. - **Expert parallelism**: to trade latency for throughput add `--ep` (see [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism)). On AMD, set `--ep` equal to `--tp`. Shared-experts fusion is automatically disabled when EP > 1; on AMD standard EP the server also disables `--enable-aiter-allreduce-fusion` automatically to preserve accuracy. - `--trust-remote-code` is required to load the MiniMax config / processor classes. diff --git a/docs_new/src/snippets/configs/MiniMaxAI/minimax-m3-benchmarks.jsx b/docs_new/src/snippets/configs/MiniMaxAI/minimax-m3-benchmarks.jsx index 659de1939..32166df7b 100644 --- a/docs_new/src/snippets/configs/MiniMaxAI/minimax-m3-benchmarks.jsx +++ b/docs_new/src/snippets/configs/MiniMaxAI/minimax-m3-benchmarks.jsx @@ -2,26 +2,27 @@ // minimax-m3.jsx cells. See _deployment.jsx for the speed/accuracy schema. // // SPEED — bench_serving --flush-cache, random isl2048/osl256, max_concurrency 64, -// CUDA graph on. B200 (tp4, MXFP8, MSA fmha_sm100 path) and H200 (tp8, bf16, -// built-in Triton sparse) are measured on PR #27944 — warm steady-state from a -// 3-run sweep (the cold-start first run, ~2x slower, is excluded). B300 / GB300 -// rows are the earlier sglang main (2026-06-11) tp4 MSA -// numbers, pending a #27944 re-measure on their own boxes. GB200 is a bare-match +// CUDA graph on. B200 (tp8, MXFP8, MSA fmha_sm100 path; re-measured 2026-06-15 +// with piecewise CUDA graph default-on) and H200 (tp8, bf16, built-in Triton +// sparse) are measured on PR #27944 — warm steady-state from a 3-run sweep (the +// B200 3-run is identical; the H200 cold-start first run, ~2x slower, is +// excluded). B300 / GB300 +// rows are the earlier 2026-06-11 tp4 MSA numbers (pre-piecewise), +// pending a #27944 re-measure on their own boxes. GB200 is a bare-match // stub (inferred-supported, not benchmarked). AMD: MI355X at 8-GPU tp8 (native // MXFP8) carries a bench_serving speed row; MI300X (MXFP8 -> block-fp8) was // accuracy-only. MI350X / MI325X inherit their same-arch sibling's recipe // (stubs). (sgl-eval does NOT measure serving throughput — TTFT/TPOT/tok-s come // from sglang.bench_serving.) // -// GSM8K — unified on a SINGLE harness: sgl-eval (github.com/sgl-project/sgl-eval) -// `run gsm8k`, full 1319-question test split, chat endpoint with --thinking -// (M3's reasoning path) + M3's recommended sampling (temp 1.0 / top_p 0.95 / -// top_k 40), symbolic grading. This is the config's Reproduce command. B200 -// (MSA path) and H200 (bf16, built-in Triton sparse) are measured on PR #27944. -// 3-run results: H200 is stable at 97.04% (std 0.0); B200's fresh-server 94.4% -// (= greedy) drifts down over sustained runs interleaved with bench (an -// MSA-under-load serving issue under investigation), so it reports the -// fresh-server value, not the drifted mean. +// GSM8K / GPQA — unified on a SINGLE harness: sgl-eval (github.com/sgl-project/sgl-eval) +// `run gsm8k` (full 1319) / `run gpqa` (GPQA Diamond 198, n-repeats 4), chat +// endpoint with --thinking (M3's reasoning path) + M3's recommended sampling +// (temp 1.0 / top_p 0.95), symbolic grading. This is the config's Reproduce command. +// H200 is stable at GSM8K 97.04% (std 0.0). B200 was re-measured 2026-06-15 on +// minimax-m3-upstream (piecewise + MSA decode fix): GSM8K 96.51% recommended / +// 96.89% greedy (stable single-run), GPQA pass@1[avg-of-4] 89.14% — the merged +// MSA decode fix resolves the earlier fresh-server-94.4%-then-drift under-load issue. // Per-platform re-measurement under sgl-eval is in progress; rows still pending // show `gsm8k_pct: null` (no GSM8K row rendered) with the legacy-harness number // kept in a comment. Legacy harnesses were NOT comparable across platforms @@ -29,14 +30,18 @@ // which is exactly why we re-measure on one harness. export const benchmarks = [ { + // B200 re-measured 2026-06-15 at tp8 on minimax-m3-upstream (piecewise CUDA + // graph default-on + AR-fusion revert/off + MSA decode fix). The earlier + // #27944 tp4 speed + GSM8K drift were pre-fix; the merged MSA decode fix + // resolves the drift (stable single-run greedy 96.89% / recommended 96.51%). match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "balanced", nodes: "single" }, sglang_version: "PR #27944", speed: [ - // bench_serving --flush-cache, MSA path; warm steady-state (3-run, cold-start run-1 excluded). + // bench_serving --flush-cache, MSA path, tp8; warm steady-state (3-run, identical). { workload: { dataset: "random", isl: 2048, osl: 256, max_concurrency: 64, num_prompts: 128 }, - ttft_ms: 749, tpot_ms: 61.5, tokens_per_sec_per_gpu: 249 }, + ttft_ms: 1580, tpot_ms: 24.1, tokens_per_sec_per_gpu: 265 }, ], - accuracy: { gsm8k_pct: 94.4 }, // #27944, sgl-eval --thinking, full 1319, recommended sampling (temp 1.0/top_p 0.95/top_k 40), MSA path; fresh-server 94.4% (greedy 94.16%; --no-thinking 88.6%). NOTE: 3 sustained runs interleaved with bench drifted 94.4->89.2->86.2 — an MSA-under-load serving issue (under investigation), not the model accuracy. + accuracy: { gpqa_pct: 89.1, gsm8k_pct: 96.5 }, // 2026-06-15, sgl-eval --thinking, recommended sampling (temp 1.0/top_p 0.95), tp8. GSM8K full 1319 = 96.51% (greedy 96.89%). GPQA Diamond 198, n-repeats 4 = pass@1[avg-of-4] 89.14% +/-1.73% (pass@4 95.45%, majority@4 93.52%). }, { // Hopper H200: bf16 build (MXFP8 is Blackwell-only) at tp8, built-in Triton @@ -52,7 +57,7 @@ export const benchmarks = [ }, { match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "balanced", nodes: "single" }, - sglang_version: "main (2026-06-11)", + sglang_version: "PR #27944", speed: [ { workload: { dataset: "random", isl: 2048, osl: 256, max_concurrency: 64 }, ttft_ms: null, tpot_ms: 32.8, tokens_per_sec_per_gpu: 365 }, @@ -63,7 +68,7 @@ export const benchmarks = [ { match: { hw: "gb200", variant: "default", quant: "mxfp8", strategy: "balanced", nodes: "single" } }, { match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "balanced", nodes: "single" }, - sglang_version: "main (2026-06-11)", + sglang_version: "PR #27944", speed: [ { workload: { dataset: "random", isl: 2048, osl: 256, max_concurrency: 64 }, ttft_ms: 4746, tpot_ms: 39.3, tokens_per_sec_per_gpu: 277 }, @@ -77,7 +82,7 @@ export const benchmarks = [ // No TTFT/TPOT reported for this run. { match: { hw: "mi355x", variant: "default", quant: "mxfp8", strategy: "balanced", nodes: "single" }, - sglang_version: "main (2026-06-11)", + sglang_version: "PR #27944", speed: [ { workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 64, num_prompts: 640 }, ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: 210 }, @@ -89,7 +94,7 @@ export const benchmarks = [ // MI300X (gfx942): MXFP8 -> block-fp8 [128,128]. { match: { hw: "mi300x", variant: "default", quant: "mxfp8", strategy: "balanced", nodes: "single" }, - sglang_version: "main (2026-06-11)", + sglang_version: "PR #27944", accuracy: { gsm8k_pct: null }, // TODO: pending sgl-eval re-measure on MI300X (legacy run_eval 1319: 92.0, triton 0.917-0.929 / aiter ~0.929) }, // MI325X (gfx942): inferred-supported from MI300X, not separately benchmarked. diff --git a/docs_new/src/snippets/configs/MiniMaxAI/minimax-m3.jsx b/docs_new/src/snippets/configs/MiniMaxAI/minimax-m3.jsx index 6d2bdfdc7..f54774953 100644 --- a/docs_new/src/snippets/configs/MiniMaxAI/minimax-m3.jsx +++ b/docs_new/src/snippets/configs/MiniMaxAI/minimax-m3.jsx @@ -64,11 +64,19 @@ sgl-eval run gsm8k \\ --model {{MODEL_NAME}} \\ --temperature 1.0 --top-p 0.95 \\ --thinking`, + gpqa_pct: +`pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run gpqa \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ + --model {{MODEL_NAME}} \\ + --temperature 1.0 --top-p 0.95 \\ + --thinking --n-repeats 4 --max-tokens 40960`, }, numPromptsByConc: { 24: 24, 64: 128 }, }, accuracyLabels: [ + ["gpqa_pct", "GPQA Diamond", "%"], ["gsm8k_pct", "GSM8K", "%"], ], @@ -173,10 +181,12 @@ sgl-eval run gsm8k \\ }, }, - // NVIDIA Blackwell: one validated single-node tp4 recipe per family. fa4 + - // page 128 + deep_gemm are the M3 SM100 auto-defaults on current main, so this - // is also the bare-launch behavior; they engage MiniMax's MSA sparse-attention - // kernel when fmha_sm100 is installed (see Configuration Tips), Triton otherwise. + // NVIDIA Blackwell: one validated single-node recipe per family — tp4 across + // B300 / GB200 / GB300, tp8 on B200. fa4 + page 128 + deep_gemm are the M3 + // SM100 auto-defaults on current main, so this is also the bare-launch + // behavior; they engage MiniMax's MSA sparse-attention kernel (fmha_sm100, + // pre-installed in the dev-minimax-m3 images; see Configuration Tips), Triton + // fallback otherwise. // AMD: tp8. MI350X/MI355X (gfx950) serve MXFP8 natively (backends auto). MI300X/ // MI325X (gfx942) need --attention-backend aiter + --moe-runner-backend triton, // and the MXFP8 weights are auto-converted to block-fp8 at load; the cold-start @@ -191,12 +201,12 @@ sgl-eval run gsm8k \\ "--model-path {{MODEL_NAME}}", "--reasoning-parser auto", "--tool-call-parser auto", - "--tp 4", + "--tp 8", "--attention-backend fa4", "--page-size 128", "--moe-runner-backend deep_gemm", "--chunked-prefill-size 8192", - "--mem-fraction-static 0.75", + "--mem-fraction-static 0.65", "--host {{HOST_IP}}", "--port {{PORT}}", ],