[Cookbook] Add DeepSeek-V4-Flash-Vision-Exp to the DeepSeek-V4 page (#37293)
This commit is contained in:
zijiexia
@@ -164,8 +164,10 @@ least one real deployment or capability boundary. Put orthogonal runtime feature
|
||||
4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell;
|
||||
set `verified: true` only on tested combos), and `modelNames` with real HF slugs,
|
||||
`dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never
|
||||
a guessed release; key by `hw`, or `hw|quant` when one quant on a shared GPU needs its own
|
||||
image), `multiNodeHints` only for fabric-specific hw (e.g. gb200).
|
||||
a guessed release; key by `hw`, or a more specific key — `hw|variant|quant`, `variant|quant`,
|
||||
`hw|quant|strategy`, `hw|quant`, resolved in that order before `hw` — when one
|
||||
variant/quant/strategy needs its own image), `multiNodeHints` only for fabric-specific hw
|
||||
(e.g. gb200).
|
||||
|
||||
### Site-wiring (do all three)
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ the full contract):
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. Hints render above **both** run modes, so keep them mode-agnostic: `docker run` flags belong in the hardware entry's `multiNodeDockerFlags` (above), which the engine puts in the command itself. |
|
||||
| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|quant` (most specific) then `hw`. Use a `hw\|quant` key only when one quant on a shared GPU needs a different image (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); otherwise key by plain `hw`. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
|
||||
| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|variant\|quant` (most specific), then `variant\|quant`, `hw\|quant\|strategy`, `hw\|quant`, `hw`. Key by plain `hw` unless a subset needs a different image: `hw\|quant` when one quant on a shared GPU needs its own build (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); the `variant` keys when one checkpoint needs its own build (e.g. a preview image for a not-yet-released variant); the `strategy` key when one tier needs one (e.g. a spec-decoding preview). **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
|
||||
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
|
||||
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. `speed` should carry `--flush-cache` (bench_serving's `random` prompts are deterministic — warm reruns hit the radix cache and inflate throughput; measure cache-cold). Optional; the button only appears when this AND `benchmarks` are present. |
|
||||
| `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `accuracyLabels` (below) + `benchmarkCommands.accuracy`. |
|
||||
|
||||
@@ -141,12 +141,14 @@ sgl-eval run gsm8k \\
|
||||
],
|
||||
},
|
||||
|
||||
// Image for `docker run` framing, keyed by `hw` (or `hw|quant`, resolved first, when one
|
||||
// quant on a shared GPU needs its own image — e.g. an FP4 dev build while FP8/BF16 use the
|
||||
// release tag). ASK the user which sglang build the recipes ran on; don't guess a supporting
|
||||
// release. Default below is :dev (nightly) — replace the tag with the user's release if they
|
||||
// give one. NVIDIA share one image; AMD uses ROCm tags. GB200/GB300/B300 may need a
|
||||
// `-cu130` (CUDA 13) tag — confirm per release.
|
||||
// Image for `docker run` framing, keyed by `hw` (or, resolved before it: `hw|variant|quant`,
|
||||
// `variant|quant`, `hw|quant|strategy`, `hw|quant` — use a more specific key only when a
|
||||
// variant/quant/strategy subset needs its own build, e.g. an FP4 dev build while FP8/BF16 use
|
||||
// the release tag, or a preview image for a not-yet-released variant). ASK the user which
|
||||
// sglang build the recipes ran on; don't guess a supporting release. Default below is :dev
|
||||
// (nightly) — replace the tag with the user's release if they give one. NVIDIA share one
|
||||
// image; AMD uses ROCm tags. GB200/GB300/B300 may need a `-cu130` (CUDA 13) tag — confirm
|
||||
// per release.
|
||||
dockerImages: {
|
||||
h100: "lmsysorg/sglang:dev",
|
||||
h200: "lmsysorg/sglang:dev",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: DeepSeek-V4
|
||||
description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, Pro, and Pro Official (0813)."
|
||||
description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, Flash Vision (Exp), Pro, and Pro Official (0813)."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
@@ -32,7 +32,7 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
|
||||
|
||||
**NVIDIA GPUs**
|
||||
|
||||
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000).
|
||||
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). The one exception is **Flash Vision (Exp)**, whose support has not shipped in a release yet: its cells use the preview image `lmsysorg/sglang:dev-dsv4-flash-vision` (the command panel picks it automatically — see the [Flash Vision notes](#vision-note)).
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
@@ -86,6 +86,10 @@ import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx";
|
||||
|
||||
<Note>
|
||||
The **Flash Vision (Exp)** variant runs on a dedicated preview Docker image, `lmsysorg/sglang:dev-dsv4-flash-vision` — its support ([sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253)) has not shipped in a release yet. The command panel's Docker mode emits that image automatically for Flash Vision cells; see the [Flash Vision notes](#vision-note).
|
||||
</Note>
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<Note>
|
||||
@@ -127,7 +131,7 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash and 0813 Pro refreshes add checkpoints with a bundled DSpark draft head:
|
||||
**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash and 0813 Pro refreshes add checkpoints with a bundled DSpark draft head, and the experimental Flash Vision checkpoint builds image understanding on top of the 0731 Flash base:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
@@ -157,6 +161,12 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flash Official (0731), with a bundled DSpark draft head; verified on 8×B200, 4×GB300, and 4×H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp">DeepSeek-V4-Flash-Vision-Exp</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>305B</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flash Vision (Exp) — experimental multimodal (image-text-to-text): the 0731 Flash base + vision encoder & aligner; verified on 4×B200 (TP=4), requires the <a href="#vision-note">preview build</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro">DeepSeek-V4-Pro</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>1.6T</strong></td>
|
||||
@@ -178,7 +188,7 @@ The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (on
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=1.0`.
|
||||
|
||||
**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · [Pro Official (0813)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro).
|
||||
**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Flash Vision (Exp)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · [Pro Official (0813)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
@@ -205,6 +215,18 @@ For the original Flash and Pro checkpoints:
|
||||
- `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
|
||||
- MTP runs on the v2 speculative path.
|
||||
|
||||
<a id="vision-note" />
|
||||
|
||||
**DeepSeek-V4-Flash-Vision-Exp (Experimental)**
|
||||
|
||||
[`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) is DeepSeek's first experimental multimodal V4 checkpoint: the 0731 Flash base plus a vision encoder and aligner, served through the same `sglang serve` flow with OpenAI-style `image_url` inputs (see [Vision](#3-5-vision-image-inputs) below). Select the **Flash Vision** variant in the Deploy panel for its recipes.
|
||||
|
||||
- **Preview build required** — support lands via [sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253) and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image `lmsysorg/sglang:dev-dsv4-flash-vision`; for a Python environment, install SGLang from that PR's branch.
|
||||
- **Verified matrix** — MMMU-Pro via sgl-eval at `temperature 1.0`, `top-p 0.95`, `--reasoning-effort max`.
|
||||
- **Engine auto-configuration** — the engine picks the `flashinfer_mxfp4` MoE runner and auto-disables shared-experts fusion for this checkpoint (its HashTopK routing rejects fused shared experts); don't pass `--enforce-shared-experts-fusion`.
|
||||
- **Chunked prefill & radix cache stay enabled** — the scheduler keeps image spans consistent automatically: chunked-prefill truncation points are span-aligned (an image span always prefills within a single extend, overshooting the chunk budget by at most one span), and a radix-cache prefix match ending deep inside an image span is re-issued from the span start.
|
||||
- **Speculative decoding** — the checkpoint bundles a DSpark head, but MTP/DSpark with image batches is not yet verified, so all Flash Vision recipes run target-only. As on the 0731/0813 checkpoints, do not pass the EAGLE flags.
|
||||
|
||||
**Shared experts fusion (Blackwell, flashinfer_mxfp4)**
|
||||
|
||||
On the Blackwell fp4 recipes (`--moe-runner-backend flashinfer_mxfp4`), the shared expert runs as a separate FP8 MLP on an alternate stream by default. Adding:
|
||||
@@ -215,7 +237,7 @@ On the Blackwell fp4 recipes (`--moe-runner-backend flashinfer_mxfp4`), the shar
|
||||
|
||||
routes it as one extra MXFP4 expert through the same trtllm-gen MoE kernel, so the whole MoE runs on a single stream (~4 fewer kernel launches and 2 fewer stream syncs per MoE layer). The shared expert is requantized from FP8 to MXFP4 at load time. Measured on GB200 tp4: gsm8k and AIME25 accuracy on par with the unfused baseline; Mean TTFT -13% to -21% and P99 ITL -15% to -53% at QPS 1-8 with neutral throughput.
|
||||
|
||||
Only for deployments without expert parallelism (e.g. the single-node low-latency recipes): with `moe_ep_size > 1` the flag is rejected at startup, unless the DeepEP/MegaMOE per-rank shared-slot path is in use.
|
||||
Only for deployments without expert parallelism (e.g. the single-node low-latency recipes): with `moe_ep_size > 1` the flag is rejected at startup, unless the DeepEP/MegaMOE per-rank shared-slot path is in use. Not applicable to [Flash Vision (Exp)](#vision-note) — the engine auto-disables the fusion on that checkpoint.
|
||||
|
||||
**Compressed attention state dtype**
|
||||
|
||||
@@ -603,6 +625,8 @@ For more details, see the [HiCache documentation](../../../docs/advanced_feature
|
||||
|
||||
Flash Official (0731) and Pro Official (0813) bundle a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731` and `deepseek-ai/DeepSeek-V4-Pro-0813`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`.
|
||||
|
||||
The experimental [Flash Vision checkpoint](#vision-note) also bundles a DSpark head, but speculative decoding with image batches is not yet verified on it — the Flash Vision recipes run target-only for now, and the Playground greys the DSpark chip out on that variant.
|
||||
|
||||
Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint.
|
||||
|
||||
<Note>
|
||||
@@ -634,3 +658,44 @@ Larger blocks can improve decode latency when acceptance stays high, but they al
|
||||
For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override.
|
||||
|
||||
DSpark currently requires CUDA, `pp_size == 1`, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
|
||||
|
||||
### 3.5 Vision (Image Inputs)
|
||||
|
||||
The experimental [`DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) checkpoint (the **Flash Vision** variant in the Deploy panel — see the [configuration notes](#vision-note)) takes images via the OpenAI-compatible `image_url` content type, as public URLs or base64 `data:` URIs; text and images mix freely in one message. Vision input works with the same server the Deploy panel produces — no extra model-specific flags needed.
|
||||
|
||||
<Accordion title="Image Understanding (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url",
|
||||
"image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
|
||||
{"type": "text", "text": "Describe this image in a few sentences."},
|
||||
],
|
||||
}],
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if getattr(message, "reasoning_content", None):
|
||||
print("=============== Thinking =================")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Content =================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Pending update...
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -80,6 +80,7 @@
|
||||
// Legacy "Mean" data is being re-measured to P50; drop once done
|
||||
// multiNodeHints optional — {[hwId]: string[]} prepended as `# ...` lines
|
||||
// dockerImages optional — `docker run` image, keyed by
|
||||
// `hw|variant|quant` then `variant|quant` then
|
||||
// `hw|quant|strategy` then `hw|quant` then `hw`;
|
||||
// falls back to `lmsysorg/sglang:dev`
|
||||
// dockerHostNetworkWhen optional — `(selection, {flags, env}) => boolean`
|
||||
@@ -780,11 +781,15 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
|
||||
let cmd;
|
||||
if (mode === "docker") {
|
||||
// Image keyed by `hw|quant|strategy` (most specific), then `hw|quant`,
|
||||
// then `hw`; `:dev` if unmapped. The strategy key covers a tier that
|
||||
// needs its own build (e.g. a spec-decoding preview image).
|
||||
// Image keyed by `hw|variant|quant` (most specific), then `variant|quant`,
|
||||
// then `hw|quant|strategy`, `hw|quant`, `hw`; `:dev` if unmapped. The
|
||||
// variant keys cover a checkpoint that needs its own build (e.g. a
|
||||
// new-variant preview image); the strategy key covers a tier that needs
|
||||
// one (e.g. a spec-decoding preview image).
|
||||
const di = config.dockerImages || {};
|
||||
const image = di[`${sel.hw}|${sel.quant}|${sel.strategy}`]
|
||||
const image = di[`${sel.hw}|${sel.variant}|${sel.quant}`]
|
||||
|| di[`${sel.variant}|${sel.quant}`]
|
||||
|| di[`${sel.hw}|${sel.quant}|${sel.strategy}`]
|
||||
|| di[`${sel.hw}|${sel.quant}`] || di[sel.hw] || "lmsysorg/sglang:dev";
|
||||
const dockerRunCommand = typeof config.dockerRunCommand === "function"
|
||||
? config.dockerRunCommand(sel)
|
||||
|
||||
@@ -1520,13 +1520,17 @@ export const Playground = ({ config }) => {
|
||||
}
|
||||
let cmd;
|
||||
if (mode === "docker") {
|
||||
// Image keyed by `hw|quant|strategy` (most specific), then `hw|quant`, then
|
||||
// `hw`; `:dev` if unmapped (matches _deployment.jsx). The strategy key covers
|
||||
// a tier that needs its own build (e.g. a spec-decoding preview image), so the
|
||||
// playground base must resolve it too or it hands back an image that cannot
|
||||
// run the command.
|
||||
// Image keyed by `hw|variant|quant` (most specific), then `variant|quant`,
|
||||
// then `hw|quant|strategy`, `hw|quant`, `hw`; `:dev` if unmapped (matches
|
||||
// _deployment.jsx). The variant keys cover a checkpoint that needs its own
|
||||
// build (e.g. a new-variant preview image) and the strategy key a tier that
|
||||
// needs one (e.g. a spec-decoding preview image), so the playground base
|
||||
// must resolve them too or it hands back an image that cannot run the
|
||||
// command.
|
||||
const di = config.dockerImages || {};
|
||||
const image = di[`${sel.hw}|${sel.quant}|${sel.strategy}`]
|
||||
const image = di[`${sel.hw}|${sel.variant}|${sel.quant}`]
|
||||
|| di[`${sel.variant}|${sel.quant}`]
|
||||
|| di[`${sel.hw}|${sel.quant}|${sel.strategy}`]
|
||||
|| di[`${sel.hw}|${sel.quant}`] || di[sel.hw] || "lmsysorg/sglang:dev";
|
||||
const dockerRunCommand = typeof config.dockerRunCommand === "function"
|
||||
? config.dockerRunCommand(sel)
|
||||
|
||||
@@ -634,4 +634,28 @@ export const benchmarks = [
|
||||
{ match: { hw: "mi355x", variant: "pro", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "mi355x", variant: "pro", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "mi355x", variant: "pro", quant: "fp8", strategy: "high-throughput", nodes: "single" } },
|
||||
// ====================================================================
|
||||
// B200 + FP4 — Flash Vision (Exp)
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b200", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "dev-dsv4-flash-vision",
|
||||
accuracy: { mmmu_pro_pct: 74.96 },
|
||||
notes: "MMMU-Pro (standard, 10-option) measured with sgl-eval on 4×B200 (TP=4) at temperature 1.0, top-p 0.95, --reasoning-effort max.",
|
||||
},
|
||||
{ match: { hw: "b200", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b200", variant: "flash-vision", quant: "fp4", strategy: "high-throughput", nodes: "single" } },
|
||||
// B300 / GB200 / GB300 / H200 / H100 — Flash Vision (Exp), all pending
|
||||
{ match: { hw: "b300", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "flash-vision", quant: "fp4", strategy: "high-throughput", nodes: "single" } },
|
||||
{ match: { hw: "gb200", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "gb200", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "gb200", variant: "flash-vision", quant: "fp4", strategy: "high-throughput", nodes: "single" } },
|
||||
{ match: { hw: "gb300", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "gb300", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "gb300", variant: "flash-vision", quant: "fp4", strategy: "high-throughput", nodes: "single" } },
|
||||
{ match: { hw: "h200", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "h200", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "h100", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" } },
|
||||
];
|
||||
|
||||
@@ -25,6 +25,7 @@ export const config = {
|
||||
variants: [
|
||||
{ id: "flash", label: "Flash", subtitle: "284B" },
|
||||
{ id: "flash-official", label: "Flash Official", subtitle: "284B · 0731" },
|
||||
{ id: "flash-vision", label: "Flash Vision", subtitle: "305B · Exp" },
|
||||
{ id: "pro", label: "Pro", subtitle: "1.6T" },
|
||||
{ id: "pro-official", label: "Pro Official", subtitle: "1.6T · 0813" },
|
||||
],
|
||||
@@ -49,6 +50,7 @@ export const config = {
|
||||
"flash|fp8": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"flash|nvfp4": "nvidia/DeepSeek-V4-Flash-NVFP4",
|
||||
"flash-official|fp4": "deepseek-ai/DeepSeek-V4-Flash-0731",
|
||||
"flash-vision|fp4": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
|
||||
"pro|fp4": "deepseek-ai/DeepSeek-V4-Pro",
|
||||
"pro|fp8": "deepseek-ai/DeepSeek-V4-Pro",
|
||||
"pro|nvfp4": "nvidia/DeepSeek-V4-Pro-NVFP4",
|
||||
@@ -144,6 +146,14 @@ sgl-eval run aime25 \\
|
||||
--n-repeats 16 --max-tokens 400000 \\
|
||||
--temperature 1.0 --top-p 1.0 --thinking \\
|
||||
--out-dir /sgl-workspace/logs \\
|
||||
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`,
|
||||
},
|
||||
mmmu_pro_pct: {
|
||||
"flash-vision":
|
||||
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
|
||||
sgl-eval run mmmu_pro \\
|
||||
--reasoning-effort max \\
|
||||
--temperature 1.0 --top-p 0.95 \\
|
||||
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`,
|
||||
},
|
||||
},
|
||||
@@ -162,6 +172,7 @@ sgl-eval run aime25 \\
|
||||
["gpqa_pct", "GPQA Diamond", "%"],
|
||||
["aime25_pct", "AIME25", "%"],
|
||||
["gsm8k_pct", "GSM8K (1-shot)", "%"],
|
||||
["mmmu_pro_pct", "MMMU-Pro (standard, 10-option)", "%"],
|
||||
],
|
||||
|
||||
// Prepended as `# ...` comments above multi-node commands.
|
||||
@@ -175,6 +186,10 @@ sgl-eval run aime25 \\
|
||||
},
|
||||
|
||||
dockerImages: {
|
||||
// Flash Vision (Exp) support has not shipped in a release yet
|
||||
// (sgl-project/sglang#37253) — until it does, the variant needs this
|
||||
// preview build on every hardware.
|
||||
"flash-vision|fp4": "lmsysorg/sglang:dev-dsv4-flash-vision",
|
||||
h100: "lmsysorg/sglang:latest",
|
||||
h200: "lmsysorg/sglang:latest",
|
||||
b200: "lmsysorg/sglang:latest",
|
||||
@@ -289,15 +304,17 @@ sgl-eval run aime25 \\
|
||||
{ id: "mtp-314", label: "EAGLE / MTP 3-1-4",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"],
|
||||
hide: { variant: ["flash-official", "pro-official"] } },
|
||||
hide: { variant: ["flash-official", "flash-vision", "pro-official"] } },
|
||||
{ id: "mtp-112", label: "EAGLE / MTP 1-1-2",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 1",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"],
|
||||
hide: { variant: ["flash-official", "pro-official"] } },
|
||||
hide: { variant: ["flash-official", "flash-vision", "pro-official"] } },
|
||||
{ id: "dspark", label: "DSpark",
|
||||
flags: ["--speculative-algorithm DSPARK"],
|
||||
hide: { variant: ["flash", "pro"] },
|
||||
disable: [
|
||||
{ when: { variant: ["flash-vision"] },
|
||||
reason: "The Flash Vision checkpoint bundles a DSpark head, but speculative decoding is not yet verified with image inputs — the cookbook recipes run target-only for now." },
|
||||
{ when: { dpAttnOn: [true] },
|
||||
reason: "DSpark is not compatible with DP Attention on the current release." },
|
||||
{ when: { hw: ["mi300x", "mi355x"] },
|
||||
@@ -2826,5 +2843,277 @@ sgl-eval run aime25 \\
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// B200 + FP4 — Flash Vision (Exp)
|
||||
//
|
||||
// DeepSeek-V4-Flash-Vision-Exp (sgl-project/sglang#37253): the 0731
|
||||
// Flash base plus a vision encoder + aligner. The checkpoint bundles a
|
||||
// DSpark head, but speculative decoding is not yet verified with image
|
||||
// batches, so every recipe runs target-only. Low-latency is the serving
|
||||
// shape the MMMU-Pro round ran on (4×B200); balanced / high-throughput
|
||||
// mirror the Flash Official recipes on the same 4-GPU topology — final
|
||||
// verification in progress.
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b200", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: true,
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "flash-vision", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [
|
||||
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
|
||||
],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend megamoe",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// B300 / GB200 / GB300 + FP4 — Flash Vision (Exp)
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b300", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "flash-vision", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [
|
||||
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
|
||||
],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend megamoe",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "flash-vision", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [
|
||||
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
|
||||
],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend megamoe",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "flash-vision", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [
|
||||
"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
|
||||
],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend megamoe",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// H200 + FP4 — Flash Vision (Exp)
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "h200", variant: "flash-vision", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--moe-runner-backend marlin",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--moe-runner-backend flashinfer_mxfp4",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// H100 + FP4 — Flash Vision (Exp)
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "h100", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
verificationStatus: "in-progress",
|
||||
warn: "DeepSeek-V4-Flash-Vision-Exp support has not shipped in an SGLang release yet (sglang PR 37253): Docker mode already points at the preview image; for Python mode install SGLang from that PR. See [Flash Vision notes](#vision-note).",
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 8",
|
||||
"--moe-runner-backend marlin",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user