minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Cookbook] DeepSeek-V4 DGX Spark: v2 image + Flash Official NVFP4 and Flash Vision FP4 cells (#37737)

Browse Source 
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jimmy Shong
2026-09-03 11:43:01 -07:00
committed by GitHub
co-authored by Claude Fable 5.1
parent abed680320
commit 2da5802bfa
2 changed files with 101 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+8 -7
View File
@@ -32,7 +32,7 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
**NVIDIA GPUs**
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). The one exception is **Flash Vision (Exp)**, whose support has not shipped in a release yet: its cells use the preview image `lmsysorg/sglang:dev-dsv4-flash-vision` (the command panel picks it automatically — see the [Flash Vision notes](#vision-note)). **DGX Spark** is the other exception: its single Flash Official FP4 cell uses the DGX Spark–only preview image `lmsysorg/sglang:dev-v4f-2dgx` (the command panel picks it automatically — see the [DGX Spark notes](#spark-note)); do not use that image on any other hardware.
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). The one exception is **Flash Vision (Exp)**, whose support has not shipped in a release yet: its cells use the preview image `lmsysorg/sglang:dev-dsv4-flash-vision` (the command panel picks it automatically — see the [Flash Vision notes](#vision-note)). **DGX Spark** is the other exception: its three cells (Flash Official FP4, Flash Official NVFP4, Flash Vision FP4) use the DGX Spark–only preview image `lmsysorg/sglang:dev-v4f-2dgx-v2` (the command panel picks it automatically — see the [DGX Spark notes](#spark-note)); do not use that image on any other hardware.
```bash Command
docker pull lmsysorg/sglang:latest
@@ -217,14 +217,15 @@ For the original Flash and Pro checkpoints:
<a id="spark-note" />
**DGX Spark (2x GB10, Flash Official FP4)**
**DGX Spark (2x GB10): Flash Official FP4 / NVFP4, Flash Vision FP4**
The **DGX Spark** row has a single recipe: **Flash Official (0731) · FP4 · Balanced · Multi-Nodes** — the 284B checkpoint does not fit one 128GB GB10, so it runs TP=2 across two DGX Sparks connected over ConnectX-7 (RoCE). Every other DGX Spark combination is greyed out on purpose.
The **DGX Spark** row has three recipes, all **Balanced · Multi-Nodes**: **Flash Official (0731) · FP4**, **Flash Official (0731) · NVFP4**, and **Flash Vision (Exp) · FP4**. None of these checkpoints fits one 128GB GB10, so every recipe runs TP=2 across two DGX Sparks connected over ConnectX-7 (RoCE). Every other DGX Spark combination is greyed out on purpose.
- **Docker image** — the cell uses `lmsysorg/sglang:dev-v4f-2dgx`, a preview build made **only for DGX Spark**: it bakes in the SM12x `b12x` MoE (W4A8) and compressed-MLA attention kernels ([#34878](https://github.com/sgl-project/sglang/pull/34878), [#35899](https://github.com/sgl-project/sglang/pull/35899), [#34018](https://github.com/sgl-project/sglang/pull/34018)) plus the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel's Docker mode — the bare Python command needs the `b12x` kernel package this image ships.
- **Docker image** — all three cells use `lmsysorg/sglang:dev-v4f-2dgx-v2`, a preview build made **only for DGX Spark** (branch `b12x-vision` @ `452239a74f`): it bakes in the SM12x `b12x` MoE (W4A8) and compressed-MLA attention kernels ([#34878](https://github.com/sgl-project/sglang/pull/34878), [#35899](https://github.com/sgl-project/sglang/pull/35899), [#34018](https://github.com/sgl-project/sglang/pull/34018)), the Flash Vision model support ([#37253](https://github.com/sgl-project/sglang/pull/37253)), the b12x image-prefill fix that lets Flash Vision serve images on SM12x, the NVFP4 MTP-layer dispatch fix, and the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel's Docker mode — the bare Python command needs the `b12x` kernel package this image ships.
- **Run the same command on both Sparks** with `--node-rank 0` / `--node-rank 1` and `--dist-init-addr` pointing at node 0 over the ConnectX-7 link. The `docker run` flags the panel emits (`--network host --ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`) are what let NCCL use RDMA; without them NCCL silently falls back to TCP and decode slows by roughly 40%.
- **Env knobs** in the cell are part of the recipe: `SGLANG_SM120_FLASHMLA_BACKEND=b12x` selects the b12x attention path, `SGLANG_B12X_MAX_TOKENS` must equal `--chunked-prefill-size`, and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` avoids unified-memory fragmentation OOMs on GB10.
- **Text only** — the image predates Flash Vision support, and the b12x kernels do not yet cover image prefill on SM12x; there is no DGX Spark cell for the Flash Vision variant.
- **Env knobs** in the cells are part of the recipe: `SGLANG_SM120_FLASHMLA_BACKEND=b12x` selects the b12x attention path, `SGLANG_B12X_MAX_TOKENS` must equal `--chunked-prefill-size`, and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` avoids unified-memory fragmentation OOMs on GB10.
- **NVFP4 (`nvidia/DeepSeek-V4-Flash-0731-NVFP4`)** — only the routed experts are NVFP4; attention, shared experts and the DSpark MTP layer stay in the checkpoint's native formats. On SM12x that means three extra flags: `--moe-runner-backend flashinfer_cutlass` (b12x's MoE is MXFP4-only and trtllm-gen kernels are sm100-only), `--speculative-moe-runner-backend b12x` (the DSpark draft's MTP experts are MXFP4 and run on b12x), and `--disable-shared-experts-fusion` (HashTopK rejects fused shared experts under the cutlass runner). Throughput and DSpark acceptance match the FP4 cell within noise.
- **Flash Vision** — images are served natively on the b12x recipe with the same flags as Flash Official (send `image_url` content on `/v1/chat/completions`, see [Vision](#3-5-vision-image-inputs)); text-only requests work unchanged. Expect roughly 15–20% lower text throughput than Flash Official on this checkpoint — its bundled DSpark head accepts fewer drafts (~3.2 vs ~3.9) — with text accuracy intact.
<a id="vision-note" />
@@ -232,7 +233,7 @@ The **DGX Spark** row has a single recipe: **Flash Official (0731) · FP4 · Bal
[`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) is DeepSeek's first experimental multimodal V4 checkpoint: the 0731 Flash base plus a vision encoder and aligner, served through the same `sglang serve` flow with OpenAI-style `image_url` inputs (see [Vision](#3-5-vision-image-inputs) below). Select the **Flash Vision** variant in the Deploy panel for its recipes.
- **Preview build required** — support lands via [sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253) and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image `lmsysorg/sglang:dev-dsv4-flash-vision`; for a Python environment, install SGLang from that PR's branch.
- **Preview build required** — support lands via [sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253) and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image `lmsysorg/sglang:dev-dsv4-flash-vision`; for a Python environment, install SGLang from that PR's branch. The DGX Spark Flash Vision cell is the exception: it uses the DGX Spark image `lmsysorg/sglang:dev-v4f-2dgx-v2` (see the [DGX Spark notes](#spark-note)).
- **Verified matrix** — MMMU-Pro via sgl-eval at `temperature 1.0`, `top-p 0.95`, `--reasoning-effort max`.
- **Engine auto-configuration** — the engine picks the `flashinfer_mxfp4` MoE runner and auto-disables shared-experts fusion for this checkpoint (its HashTopK routing rejects fused shared experts); don't pass `--enforce-shared-experts-fusion`.
- **Chunked prefill & radix cache stay enabled** — the scheduler keeps image spans consistent automatically: chunked-prefill truncation points are span-aligned (an image span always prefills within a single extend, overshooting the chunk budget by at most one span), and a radix-cache prefix match ending deep inside an image span is re-issued from the span start.
docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx
+93 -14
View File
@@ -198,10 +198,14 @@ sgl-eval run mmmu_pro \\
"flash-vision|fp4": "lmsysorg/sglang:dev-dsv4-flash-vision",
// DGX Spark ONLY. A dedicated preview build for the 2x GB10 pair: it bakes
// in the SM12x b12x MoE/attention kernels (sgl-project/sglang#34878,
// #35899, #34018) and CuTeDSL/NCCL pins the GB10 recipe needs, none of
// which are in `latest`. It is not built for, and must not be used on, any
// other hardware — every other row keeps its own image.
"dgx-spark|flash-official|fp4": "lmsysorg/sglang:dev-v4f-2dgx",
// #35899, #34018), the b12x dual-cache image-prefill fix that makes Flash
// Vision serve images on SM12x, the NVFP4 MTP-layer dispatch fix, and the
// CuTeDSL/NCCL pins the GB10 recipe needs — none of which are in `latest`.
// v2 = branch b12x-vision @ 452239a74f. It is not built for, and must not
// be used on, any other hardware — every other row keeps its own image.
"dgx-spark|flash-official|fp4": "lmsysorg/sglang:dev-v4f-2dgx-v2",
"dgx-spark|flash-official|nvfp4": "lmsysorg/sglang:dev-v4f-2dgx-v2",
"dgx-spark|flash-vision|fp4": "lmsysorg/sglang:dev-v4f-2dgx-v2",
// NVFP4 checkpoints crash at weight load on v0.5.18 (the MXFP4-packed MTP
// layer's FP8 delegate needs the #36275 guard, merged 2026-08-26) — route
// every NVFP4 cell to the nightly until a release contains that fix.
@@ -3030,23 +3034,98 @@ sgl-eval run mmmu_pro \\
},
// ====================================================================
// DGX Spark (GB10 / SM121) — Flash Official FP4, 2-node TP=2, Balanced
// DGX Spark (GB10 / SM121) — 2-node TP=2, Balanced: Flash Official FP4,
// Flash Official NVFP4, Flash Vision FP4
// ====================================================================
// One cell only: the verified GB10 recipe. It runs the SM12x b12x MoE
// (W4A8) + b12x compressed-MLA attention with DSpark, split TP=2 across two
// DGX Sparks over ConnectX-7 RoCE. Verified end to end on 2x DGX Spark with
// the `lmsysorg/sglang:dev-v4f-2dgx` image (GSM8K 96%, AgentX c1/c2 clean).
// Every other DGX Spark combination (other variants / quants / strategies /
// single node) is intentionally absent and greys out: a single 128GB GB10
// cannot hold the checkpoint, and the b12x kernels are text-only today
// (image prefill for Flash Vision is unsupported on SM12x).
// Three cells, all on the GB10 recipe: SM12x b12x compressed-MLA attention
// with DSpark, split TP=2 across two DGX Sparks over ConnectX-7 RoCE, image
// `lmsysorg/sglang:dev-v4f-2dgx-v2` (b12x-vision @ 452239a74f). Every other
// DGX Spark combination (other strategies / single node / Flash / Pro) is
// intentionally absent and greys out: a single 128GB GB10 cannot hold the
// checkpoints, and only Balanced has been run.
// - Flash Official FP4: b12x W4A8 MoE (verified on the v2 image: GSM8K
// 96.5%; earlier same-recipe runs: ~224 tok/s plateau, AgentX c1/c2 clean,
// decode microbench at parity with the qualified stack).
// - Flash Official NVFP4: the NVFP4 routed experts need the flashinfer
// cutlass runner (b12x's MoE is MXFP4-only; trtllm-gen is sm100-only); the
// DSpark draft's MTP experts stay MXFP4 and run on b12x
// (--speculative-moe-runner-backend b12x); HashTopK rejects fused shared
// experts under the cutlass runner (--disable-shared-experts-fusion).
// Verified on the v2 image: GSM8K 97.5%, DSpark accept 3.96, throughput
// at parity with the FP4 cell.
// - Flash Vision FP4: same flags as Flash Official; images are served
// natively on b12x (dual-cache prefill gate fix in the v2 image).
// Verified on the v2 image with the cookbook Reproduce commands:
// sgl-eval gsm8k 97.5% (200 q), sgl-eval mmmu_pro 85% / 0% errors
// (20-q subset, --reasoning-effort max, temp 1.0, top-p 0.95).
// Env: b12x attention + FP8 wo_a opt-in + MHC post/pre fusion are the GB10
// tuning knobs; SGLANG_B12X_MAX_TOKENS must track --chunked-prefill-size;
// expandable_segments avoids unified-memory fragmentation OOMs.
{
match: { hw: "dgx-spark", variant: "flash-official", quant: "fp4", strategy: "balanced", nodes: "multi-2" },
verified: true,
warn: "The Docker image lmsysorg/sglang:dev-v4f-2dgx is a DGX Spark-only preview build (2x GB10, TP=2 over ConnectX-7) — do not use it on other hardware. Use Docker mode: the bare Python command needs the b12x kernel package this image ships. See [DGX Spark notes](#spark-note).",
warn: "The Docker image lmsysorg/sglang:dev-v4f-2dgx-v2 is a DGX Spark-only preview build (2x GB10, TP=2 over ConnectX-7) — do not use it on other hardware. Use Docker mode: the bare Python command needs the b12x kernel package this image ships. See [DGX Spark notes](#spark-note).",
env: [
"SGLANG_SM120_FLASHMLA_BACKEND=b12x",
"B12X_MLA_SM120_DSV4_H16_NATIVE=1",
"SGLANG_OPT_FUSE_MHC_POST_PRE=1",
"SGLANG_OPT_FP8_WO_A_GEMM=1",
"SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK=1",
"SGLANG_B12X_MAX_TOKENS=8192",
"PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True",
],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 2",
"--moe-runner-backend b12x",
"--speculative-algorithm DSPARK",
"--chunked-prefill-size 8192",
"--context-length 327680",
"--mem-fraction-static 0.80",
"--swa-full-tokens-ratio 0.2",
"--cuda-graph-max-bs-decode 32",
"--max-running-requests 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "dgx-spark", variant: "flash-official", quant: "nvfp4", strategy: "balanced", nodes: "multi-2" },
verified: true,
warn: "The Docker image lmsysorg/sglang:dev-v4f-2dgx-v2 is a DGX Spark-only preview build — do not use it on other hardware, and use Docker mode. NVFP4 on DGX Spark needs the three extra MoE flags shown (cutlass runner for the NVFP4 experts, b12x for the DSpark draft's MXFP4 MTP experts, shared-experts fusion off). See [DGX Spark notes](#spark-note).",
env: [
"SGLANG_SM120_FLASHMLA_BACKEND=b12x",
"B12X_MLA_SM120_DSV4_H16_NATIVE=1",
"SGLANG_OPT_FUSE_MHC_POST_PRE=1",
"SGLANG_OPT_FP8_WO_A_GEMM=1",
"SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK=1",
"SGLANG_B12X_MAX_TOKENS=8192",
"PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True",
],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 2",
"--moe-runner-backend flashinfer_cutlass",
"--speculative-moe-runner-backend b12x",
"--disable-shared-experts-fusion",
"--speculative-algorithm DSPARK",
"--chunked-prefill-size 8192",
"--context-length 327680",
"--mem-fraction-static 0.80",
"--swa-full-tokens-ratio 0.2",
"--cuda-graph-max-bs-decode 32",
"--max-running-requests 32",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "dgx-spark", variant: "flash-vision", quant: "fp4", strategy: "balanced", nodes: "multi-2" },
verified: true,
warn: "The Docker image lmsysorg/sglang:dev-v4f-2dgx-v2 is a DGX Spark-only preview build — do not use it on other hardware, and use Docker mode. Images go in as OpenAI image_url content on /v1/chat/completions (see Vision below); text-only requests work unchanged. See [DGX Spark notes](#spark-note).",
env: [
"SGLANG_SM120_FLASHMLA_BACKEND=b12x",
"B12X_MLA_SM120_DSV4_H16_NATIVE=1",