diff --git a/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx b/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx index ce5c84675..88fb58b8e 100644 --- a/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx +++ b/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx @@ -1,6 +1,6 @@ --- title: DeepSeek-V4 -description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for the Flash (284B) and Pro (1.6T) Mixture-of-Experts models." +description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, and Pro." tag: NEW --- @@ -127,7 +127,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; ## 1. Model Introduction -**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. It ships as two Instruct repos (one per variant) plus matching Base repos: +**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash refresh adds a checkpoint with a bundled DSpark draft head: @@ -151,6 +151,12 @@ import { Playground } from "/src/snippets/_playground.jsx"; + + + + + + @@ -160,13 +166,13 @@ import { Playground } from "/src/snippets/_playground.jsx";
13B single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); RTX PRO 6000 (TP=2); H100 (TP=8)
DeepSeek-V4-Flash-073130413BFlash Official (0731), with a bundled DSpark draft head; verified for low-latency serving on 4×GB300
DeepSeek-V4-Pro 1.6T
-Both Instruct repos ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling. +The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling. **Highlights:** hybrid CSA + HCA attention (~27% inference FLOPs / ~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, **1M-token context** (32T+ pre-training tokens), three reasoning modes (*Non-think* / *Think High* / *Think Max* — use ≥ 384K context for Think Max), and a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar. **Recommended generation:** `temperature=1.0`, `top_p=1.0`. -**Resources:** HuggingFace · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro). +**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro). ## 2. Configuration Tips @@ -178,7 +184,11 @@ Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_D The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised. -**MTP (Multi-Token Prediction, EAGLE)** +**Speculative decoding** + +The original Flash and Pro recipes use EAGLE. Flash Official (0731) uses the bundled DSpark draft head; see [DSpark](#3-4-dspark-speculative-decoding) for its launch and tuning notes. + +For the original Flash and Pro checkpoints: - `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1. - `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch. @@ -549,3 +559,35 @@ For AMD devices, The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow. For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache). + +### 3.4 DSpark (Speculative Decoding) + +Flash Official (0731) bundles a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`. + +Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint. + +The verified 4×GB300 FP4 low-latency command is: + +```bash Command +sglang serve \ + --trust-remote-code \ + --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \ + --tp 4 \ + --moe-runner-backend flashinfer_mxfp4 \ + --speculative-algorithm DSPARK \ + --mem-fraction-static 0.90 \ + --chunked-prefill-size 4096 \ + --swa-full-tokens-ratio 0.1 \ + --host 0.0.0.0 \ + --port 30000 +``` + +Keep `--mem-fraction-static 0.90` on this topology to leave enough headroom for the batch-256 verify graph. The first cold start can take 10–15 minutes while FlashInfer autotunes and SGLang captures the draft and verify graphs; later starts reuse the cache. This path is verified end-to-end on 4×GB300 with SGLang v0.5.16. + +**Tune proposed draft tokens.** `--speculative-dspark-block-size N` asks DSpark to propose `N` tokens per step; the target verifies a window of `N + 1`. If the flag is omitted, SGLang reads the value from the checkpoint. The current 0731 checkpoint resolves to five proposed tokens, which is the verified default. Use the **DSpark Proposed Draft Tokens** slider in the [Playground](#playground) to sweep one through five. + +Larger blocks can improve decode latency when acceptance stays high, but they also increase verification work and graph memory. Start from the checkpoint default, then sweep downward under the real prompt-length and concurrency distribution. The gain is usually largest for short interactive traffic and narrows as prefill dominates. Track P50/P99 TTFT and TPOT, total throughput, accepted length, GPU memory, and stop rate rather than choosing from acceptance alone. + +For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override. + +DSpark currently requires CUDA, `pp_size == 1`, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; turn DSpark off before selecting a prefill or decode role. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates. diff --git a/docs_new/src/snippets/_playground.jsx b/docs_new/src/snippets/_playground.jsx index cfb2020fb..666a40756 100644 --- a/docs_new/src/snippets/_playground.jsx +++ b/docs_new/src/snippets/_playground.jsx @@ -792,6 +792,7 @@ export const Playground = ({ config }) => { || head === "--speculative-num-steps" || head === "--speculative-eagle-topk" || head === "--speculative-num-draft-tokens" + || head === "--speculative-dspark-block-size" || head === "--speculative-ngram-max-bfs-breadth"; }); if (baseSpec.length === 0) return "off"; @@ -803,18 +804,21 @@ export const Playground = ({ config }) => { return "current"; }, - apply: ({ flags, env, value, fc, h, derived }) => { + apply: ({ flags, env, value, fc, sel, h, derived }) => { if (value === "current") return { flags, env }; // No-op when the pick already matches base (preserves flag position). if (derived && value === derived) return { flags, env }; const picked = (fc.options || []).find((p) => p.id === value); - if (picked && h.evaluateChip(picked, - { dpAttnOn: h.hasFlag(flags, "--enable-dp-attention") }).disabled) { + if (picked && h.evaluateChip(picked, { + ...sel, + dpAttnOn: h.hasFlag(flags, "--enable-dp-attention"), + }).disabled) { return { flags, env }; } flags = h.stripFlagsByFirstToken(flags, [ "--speculative-algorithm", "--speculative-num-steps", "--speculative-eagle-topk", "--speculative-num-draft-tokens", + "--speculative-dspark-block-size", "--speculative-ngram-max-bfs-breadth", ]); const preset = (fc.options || []).find((p) => p.id === value); @@ -873,6 +877,10 @@ export const Playground = ({ config }) => { "--disaggregation-mode", "--disaggregation-transfer-backend", "--disaggregation-ib-device", "--disaggregation-bootstrap-port", ]); + const specAlgorithm = (h.findFlagArg(flags, "--speculative-algorithm") || "").toUpperCase(); + if ((fc.incompatibleSpeculativeAlgorithms || []).includes(specAlgorithm)) { + return { flags, env }; + } const backends = fc.transferBackends || []; // A config that omits `modes` has the role on the Deploy panel instead; // this card then only tunes the transport for whatever role is selected. @@ -922,7 +930,12 @@ export const Playground = ({ config }) => { return { flags, env }; }, - getRenderHints: (value) => { + getRenderHints: (value, fc, context) => { + const specAlgorithm = (context.h.findFlagArg( + context.flags, "--speculative-algorithm") || "").toUpperCase(); + if ((fc.incompatibleSpeculativeAlgorithms || []).includes(specAlgorithm)) { + return null; + } if (value.mode === "prefill" || value.mode === "decode") { return { pdMode: value.mode }; } @@ -1318,11 +1331,15 @@ export const Playground = ({ config }) => { const value = allDeltas[axisId]; if (value === undefined) continue; const derived = derivedMap ? derivedMap[axisId] : null; - const out = handler.apply({ flags, env, value, fc, sel, h: helpers, derived }); + const specAlgorithm = (findFlagArg( + flags, "--speculative-algorithm") || "").toUpperCase() || null; + const liveSel = { ...sel, specAlgorithm }; + const out = handler.apply({ flags, env, value, fc, sel: liveSel, h: helpers, derived }); flags = out.flags; env = out.env; if (handler.getRenderHints) { - const hints = handler.getRenderHints(value, fc) || {}; + const hints = handler.getRenderHints( + value, fc, { flags, env, sel: liveSel, h: helpers }) || {}; if (hints.pdMode) pdMode = hints.pdMode; } } @@ -2005,12 +2022,20 @@ export const Playground = ({ config }) => { ? attnDelta.cpStrategy : (attnDerived.cpStrategy !== undefined ? attnDerived.cpStrategy : null)) || "interleave"; + const constraintEffective = baseCell + ? applyAllDeltas(baseCell.flags, baseCell.env, deltas, base, derivedMap) + : null; const pdCardOwnsMode = ((pgFeatures.pdDisagg && pgFeatures.pdDisagg.modes) || []).length > 0; const pdMode = pdCardOwnsMode - ? ((deltas.pdDisagg && deltas.pdDisagg.mode) || "off") + ? ((constraintEffective && constraintEffective.pdMode) || "off") : (base.pdMode || "off"); + const specAlgorithm = constraintEffective + ? ((findFlagArg(constraintEffective.flags, + "--speculative-algorithm") || "").toUpperCase() || null) + : null; const constraintBase = { ...base, dpAttnOn, cpOn, cpStrategy, cpSizeTarget, effTp, pdMode, + specAlgorithm, }; let baseCommand = ""; @@ -2069,12 +2094,20 @@ export const Playground = ({ config }) => { const matchedSiblingCell = (matchedCell && DIMENSIONS.some((d) => matchedCell.match[d] !== base[d])) ? matchedCell : null; - // MTP hint on the EFFECTIVE (post-override) command — fires when the user - // toggles speculative decoding on without setting --max-running-requests - // (NOT keyed on strategy). Mirrors the Deploy panel's hint. - const pgMtpHint = - pgFlagsLatest.some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") && + const pgSpecAlgoFlag = pgFlagsLatest.find( + (f) => f.split(/[\s=]/)[0] === "--speculative-algorithm"); + const pgSpecHint = + !!pgSpecAlgoFlag && !pgFlagsLatest.some((f) => f.split(/[\s=]/)[0] === "--max-running-requests"); + const specAlgoLabels = { + EAGLE: "MTP", EAGLE3: "MTP", FROZEN_KV_MTP: "MTP", + DSPARK: "DSpark", DFLASH: "DFlash", NGRAM: "N-gram", + STANDALONE: "standalone draft", + }; + const pgSpecAlgoValue = pgSpecAlgoFlag + ? (pgSpecAlgoFlag.split(/[\s=]/).filter(Boolean)[1] || "") : ""; + const pgSpecAlgoName = specAlgoLabels[pgSpecAlgoValue.toUpperCase()] + || pgSpecAlgoValue || "MTP"; // Interleave prefill-CP + DP-Attention hint on the EFFECTIVE command: // deliberately allowed (combined support is planned upstream), but current @@ -2348,9 +2381,9 @@ export const Playground = ({ config }) => { )) : "# No verified base cell at the current Deployment selection.\n# Pick a supported hardware/variant in the Deployment panel to populate the playground base."} - {pgMtpHint && ( + {pgSpecHint && (
- ⚠️ Speculative decoding (MTP) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. + ⚠️ Speculative decoding ({pgSpecAlgoName}) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency.
)} {pgCpDpHint && ( diff --git a/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx b/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx index 5f2e08bcd..511d5bf4e 100644 --- a/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx +++ b/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx @@ -1,6 +1,6 @@ // DeepSeek-V4 per-cell benchmark numbers, keyed by the same `match` tuple as // deepseek-v4.jsx cells. See _deployment.jsx for the speed/accuracy schema. -// Measured on sglang v0.5.15 / v0.5.15.post1 (per-cell sglang_version). +// Measured on sglang v0.5.15 / v0.5.15.post1 / v0.5.16 (per-cell sglang_version). // tokens_per_sec_per_gpu is total (input+output) tok/s/GPU = output/GPU × (isl+osl)/osl. export const benchmarks = [ // ==================================================================== @@ -236,6 +236,37 @@ export const benchmarks = [ // ==================================================================== // GB300 + FP4 // ==================================================================== + { + match: { hw: "gb300", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + sglang_version: "0.5.16", + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 }, + ttft_ms: 468.93, tpot_ms: 1.31, tokens_per_sec_per_gpu: 930 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 }, + ttft_ms: 726.81, tpot_ms: 8.84, tokens_per_sec_per_gpu: 2711 }, + ], + accuracy: { gpqa_pct: 87.03, aime25_pct: 96.25, gsm8k_pct: 97.04 }, + }, + { + match: { hw: "gb300", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "single" }, + sglang_version: "0.5.16", + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 }, + ttft_ms: 1292.88, tpot_ms: 46.03, tokens_per_sec_per_gpu: 2678 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 }, + ttft_ms: 5861.87, tpot_ms: 103.54, tokens_per_sec_per_gpu: 5030 }, + ], + }, + { + match: { hw: "gb300", variant: "flash-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, + sglang_version: "0.5.16", + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 }, + ttft_ms: 156841.34, tpot_ms: 106.18, tokens_per_sec_per_gpu: 5520 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 }, + ttft_ms: 410259.84, tpot_ms: 105.33, tokens_per_sec_per_gpu: 5461 }, + ], + }, { match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, sglang_version: "0.5.15.post1", diff --git a/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx b/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx index f2a11ba55..57f832b57 100644 --- a/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx +++ b/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx @@ -22,6 +22,7 @@ export const config = { variants: [ { id: "flash", label: "Flash", subtitle: "284B" }, + { id: "flash-official", label: "Flash Official", subtitle: "284B · 0731" }, { id: "pro", label: "Pro", subtitle: "1.6T" }, ], quantizations: [ @@ -44,6 +45,7 @@ export const config = { "flash|fp4": "deepseek-ai/DeepSeek-V4-Flash", "flash|fp8": "deepseek-ai/DeepSeek-V4-Flash", "flash|nvfp4": "nvidia/DeepSeek-V4-Flash-NVFP4", + "flash-official|fp4": "deepseek-ai/DeepSeek-V4-Flash-0731", "pro|fp4": "deepseek-ai/DeepSeek-V4-Pro", "pro|fp8": "deepseek-ai/DeepSeek-V4-Pro", "pro|nvfp4": "nvidia/DeepSeek-V4-Pro-NVFP4", @@ -90,6 +92,14 @@ sgl-eval run gsm8k \\ gpqa_pct: { flash: `# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run gpqa \\ + --model {{MODEL_NAME}} --api-key \\ + --n-repeats 16 --max-tokens 200000 \\ + --temperature 1.0 --top-p 1.0 --thinking \\ + --out-dir /sgl-workspace/logs \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`, + "flash-official": +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval sgl-eval run gpqa \\ --model {{MODEL_NAME}} --api-key \\ --n-repeats 16 --max-tokens 200000 \\ @@ -106,6 +116,14 @@ sgl-eval run gpqa \\ --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`, }, aime25_pct: { + "flash-official": +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run aime25 \\ + --model {{MODEL_NAME}} --api-key \\ + --n-repeats 16 --max-tokens 200000 \\ + --temperature 1.0 --top-p 1.0 --thinking \\ + --out-dir /sgl-workspace/logs \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`, flash: `# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval sgl-eval run aime25 \\ @@ -268,10 +286,21 @@ sgl-eval run aime25 \\ { id: "off", label: "Off (greedy)" }, { id: "mtp-314", label: "EAGLE / MTP 3-1-4", flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3", - "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] }, + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"], + hide: { variant: ["flash-official"] } }, { id: "mtp-112", label: "EAGLE / MTP 1-1-2", flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 1", - "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"] }, + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"], + hide: { variant: ["flash-official"] } }, + { id: "dspark", label: "DSpark", + flags: ["--speculative-algorithm DSPARK"], + hide: { variant: ["flash", "pro"] }, + disable: [ + { when: { dpAttnOn: [true] }, + reason: "DSpark is not compatible with DP Attention on the current release." }, + { when: { hw: ["mi300x", "mi355x"] }, + reason: "DSpark currently requires CUDA." }, + ] }, { id: "ngram", label: "NGRAM", flags: ["--speculative-algorithm NGRAM", "--speculative-num-draft-tokens 16", @@ -285,6 +314,8 @@ sgl-eval run aime25 \\ // ----- Card 5: "PD Disaggregation" ----- pdDisagg: { + showWhen: (base) => base.specAlgorithm !== "DSPARK", + incompatibleSpeculativeAlgorithms: ["DSPARK"], modes: [ { id: "off", label: "Off" }, { id: "prefill", label: "Prefill role" }, @@ -383,12 +414,47 @@ sgl-eval run aime25 \\ ], defaultHostRatio: 10, }, + + flagSelects: [ + { + id: "dsparkDraftTokens", + title: "DSpark Proposed Draft Tokens", + showWhen: (base) => base.variant === "flash-official" && base.specAlgorithm === "DSPARK", + control: "slider", + stripPrefixes: ["--speculative-dspark-block-size"], + options: [ + { id: "auto", label: "Checkpoint default" }, + { id: "1", label: "1", flags: ["--speculative-dspark-block-size 1"] }, + { id: "2", label: "2", flags: ["--speculative-dspark-block-size 2"] }, + { id: "3", label: "3", flags: ["--speculative-dspark-block-size 3"] }, + { id: "4", label: "4", flags: ["--speculative-dspark-block-size 4"] }, + { id: "5", label: "5", flags: ["--speculative-dspark-block-size 5"] }, + ], + }, + ], }, cells: [ // ==================================================================== // B200 + FP4 // ==================================================================== + { + match: { hw: "b200", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend flashinfer_mxfp4", + "--speculative-algorithm DSPARK", + "--chunked-prefill-size 4096", + "--disable-flashinfer-autotune", + "--swa-full-tokens-ratio 0.1", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "b200", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: true, @@ -409,6 +475,22 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "b200", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "single" }, + verified: false, + env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "b200", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" }, verified: true, @@ -429,6 +511,23 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "b200", variant: "flash-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [ + "SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend megamoe", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "b200", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" }, verified: true, @@ -516,6 +615,23 @@ sgl-eval run aime25 \\ ], }, + { + match: { hw: "b300", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend flashinfer_mxfp4", + "--speculative-algorithm DSPARK", + "--chunked-prefill-size 4096", + "--disable-flashinfer-autotune", + "--swa-full-tokens-ratio 0.1", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "b300", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: true, @@ -536,6 +652,22 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "b300", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "single" }, + verified: false, + env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "b300", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" }, verified: true, @@ -556,6 +688,23 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "b300", variant: "flash-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [ + "SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend megamoe", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "b300", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" }, verified: true, @@ -735,6 +884,23 @@ sgl-eval run aime25 \\ // ==================================================================== // GB200 + FP4 // ==================================================================== + { + match: { hw: "gb200", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend flashinfer_mxfp4", + "--speculative-algorithm DSPARK", + "--chunked-prefill-size 4096", + "--disable-flashinfer-autotune", + "--swa-full-tokens-ratio 0.1", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: true, @@ -755,6 +921,22 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "gb200", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "single" }, + verified: false, + env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" }, verified: true, @@ -775,6 +957,23 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "gb200", variant: "flash-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [ + "SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend megamoe", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" }, verified: true, @@ -914,6 +1113,23 @@ sgl-eval run aime25 \\ // ==================================================================== // GB300 + FP4 // ==================================================================== + { + match: { hw: "gb300", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend flashinfer_mxfp4", + "--speculative-algorithm DSPARK", + "--mem-fraction-static 0.90", + "--chunked-prefill-size 4096", + "--swa-full-tokens-ratio 0.1", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: true, @@ -933,6 +1149,22 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "gb300", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" }, verified: true, @@ -953,6 +1185,23 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "gb300", variant: "flash-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [ + "SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend megamoe", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" }, verified: true, @@ -1216,6 +1465,20 @@ sgl-eval run aime25 \\ ], }, + { + match: { hw: "h200", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend marlin", + "--speculative-algorithm DSPARK", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "h200", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: true, @@ -1233,6 +1496,20 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "h200", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend flashinfer_mxfp4", + "--speculative-algorithm DSPARK", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "h200", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" }, verified: true, @@ -1250,6 +1527,19 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "h200", variant: "flash-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend marlin", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "h200", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" }, verified: true, @@ -1317,6 +1607,20 @@ sgl-eval run aime25 \\ // ==================================================================== // H100 + FP4 (Marlin runner) // ==================================================================== + { + match: { hw: "h100", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--moe-runner-backend marlin", + "--speculative-algorithm DSPARK", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "h100", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: true, @@ -1334,6 +1638,20 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "h100", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--moe-runner-backend marlin", + "--speculative-algorithm DSPARK", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "h100", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" }, verified: true, @@ -1351,6 +1669,19 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "h100", variant: "flash-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--moe-runner-backend marlin", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "h100", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" }, verified: true, @@ -1422,6 +1753,21 @@ sgl-eval run aime25 \\ // ==================================================================== // RTX PRO 6000 (SM120 / Blackwell Desktop) — Flash + low-latency only // ==================================================================== + { + match: { hw: "rtx6000", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 2", + "--moe-runner-backend flashinfer_mxfp4", + "--mem-fraction-static 0.92", + "--cuda-graph-max-bs-decode 32", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "rtx6000", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: true, @@ -1536,6 +1882,29 @@ sgl-eval run aime25 \\ }, // ---------- MI355X (288GB) — Flash FP4 ---------- + { + match: { hw: "mi355x", variant: "flash-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [ + "SGLANG_USE_ROCM700A=0", + "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", + "AITER_BF16_FP8_MOE_BOUND=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--attention-backend dsv4", + "--page-size 256", + "--mem-fraction-static 0.90", + "--swa-full-tokens-ratio 0.15", + "--disable-shared-experts-fusion", + "--kv-cache-dtype fp8_e4m3", + "--chunked-prefill-size 8192", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "mi355x", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: true, @@ -1563,6 +1932,36 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "mi355x", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "single" }, + verified: false, + env: [ + "SGLANG_USE_ROCM700A=0", + "SGLANG_SHARED_EXPERT_TP1=1", + "SGLANG_DP_SHARED_EXPERT_LOCAL=1", + "SGLANG_DP_USE_GATHERV=1", + "SGLANG_DP_USE_REDUCE_SCATTER=1", + "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", + "AITER_BF16_FP8_MOE_BOUND=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--enable-two-batch-overlap", + "--attention-backend dsv4", + "--page-size 256", + "--mem-fraction-static 0.90", + "--swa-full-tokens-ratio 0.15", + "--disable-shared-experts-fusion", + "--kv-cache-dtype fp8_e4m3", + "--chunked-prefill-size 65536", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "mi355x", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" }, verified: true, @@ -1597,6 +1996,36 @@ sgl-eval run aime25 \\ "--port {{PORT}}", ], }, + { + match: { hw: "mi355x", variant: "flash-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [ + "SGLANG_USE_ROCM700A=0", + "SGLANG_SHARED_EXPERT_TP1=1", + "SGLANG_DP_SHARED_EXPERT_LOCAL=1", + "SGLANG_DP_USE_GATHERV=1", + "SGLANG_DP_USE_REDUCE_SCATTER=1", + "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", + "AITER_BF16_FP8_MOE_BOUND=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--enable-two-batch-overlap", + "--attention-backend dsv4", + "--page-size 256", + "--mem-fraction-static 0.90", + "--swa-full-tokens-ratio 0.15", + "--disable-shared-experts-fusion", + "--kv-cache-dtype fp8_e4m3", + "--chunked-prefill-size 65536", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, { match: { hw: "mi355x", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" }, verified: true,