minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Cookbook] DeepSeek-V4 DGX Spark: v2 image + Flash Official NVFP4 and Flash Vision FP4 cells (#37737)

Browse Source 
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Jimmy Shong
2026-09-03 11:43:01 -07:00
committed by GitHub
co-authored by Claude Fable 5.1
parent abed680320
commit 2da5802bfa
2 changed files with 101 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+8 -7
View File
@@ -32,7 +32,7 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
**NVIDIA GPUs**
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). The one exception is **Flash Vision (Exp)**, whose support has not shipped in a release yet: its cells use the preview image `lmsysorg/sglang:dev-dsv4-flash-vision` (the command panel picks it automatically — see the [Flash Vision notes](#vision-note)). **DGX Spark** is the other exception: its single Flash Official FP4 cell uses the DGX Spark–only preview image `lmsysorg/sglang:dev-v4f-2dgx` (the command panel picks it automatically — see the [DGX Spark notes](#spark-note)); do not use that image on any other hardware.
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). The one exception is **Flash Vision (Exp)**, whose support has not shipped in a release yet: its cells use the preview image `lmsysorg/sglang:dev-dsv4-flash-vision` (the command panel picks it automatically — see the [Flash Vision notes](#vision-note)). **DGX Spark** is the other exception: its three cells (Flash Official FP4, Flash Official NVFP4, Flash Vision FP4) use the DGX Spark–only preview image `lmsysorg/sglang:dev-v4f-2dgx-v2` (the command panel picks it automatically — see the [DGX Spark notes](#spark-note)); do not use that image on any other hardware.
```bash Command
docker pull lmsysorg/sglang:latest
@@ -217,14 +217,15 @@ For the original Flash and Pro checkpoints:
<a id="spark-note" />
**DGX Spark (2x GB10, Flash Official FP4)**
**DGX Spark (2x GB10): Flash Official FP4 / NVFP4, Flash Vision FP4**
The **DGX Spark** row has a single recipe: **Flash Official (0731) · FP4 · Balanced · Multi-Nodes** — the 284B checkpoint does not fit one 128GB GB10, so it runs TP=2 across two DGX Sparks connected over ConnectX-7 (RoCE). Every other DGX Spark combination is greyed out on purpose.
The **DGX Spark** row has three recipes, all **Balanced · Multi-Nodes**: **Flash Official (0731) · FP4**, **Flash Official (0731) · NVFP4**, and **Flash Vision (Exp) · FP4**. None of these checkpoints fits one 128GB GB10, so every recipe runs TP=2 across two DGX Sparks connected over ConnectX-7 (RoCE). Every other DGX Spark combination is greyed out on purpose.
- **Docker image** — the cell uses `lmsysorg/sglang:dev-v4f-2dgx`, a preview build made **only for DGX Spark**: it bakes in the SM12x `b12x` MoE (W4A8) and compressed-MLA attention kernels ([#34878](https://github.com/sgl-project/sglang/pull/34878), [#35899](https://github.com/sgl-project/sglang/pull/35899), [#34018](https://github.com/sgl-project/sglang/pull/34018)) plus the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel's Docker mode — the bare Python command needs the `b12x` kernel package this image ships.
- **Docker image** — all three cells use `lmsysorg/sglang:dev-v4f-2dgx-v2`, a preview build made **only for DGX Spark** (branch `b12x-vision` @ `452239a74f`): it bakes in the SM12x `b12x` MoE (W4A8) and compressed-MLA attention kernels ([#34878](https://github.com/sgl-project/sglang/pull/34878), [#35899](https://github.com/sgl-project/sglang/pull/35899), [#34018](https://github.com/sgl-project/sglang/pull/34018)), the Flash Vision model support ([#37253](https://github.com/sgl-project/sglang/pull/37253)), the b12x image-prefill fix that lets Flash Vision serve images on SM12x, the NVFP4 MTP-layer dispatch fix, and the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel's Docker mode — the bare Python command needs the `b12x` kernel package this image ships.
- **Run the same command on both Sparks** with `--node-rank 0` / `--node-rank 1` and `--dist-init-addr` pointing at node 0 over the ConnectX-7 link. The `docker run` flags the panel emits (`--network host --ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`) are what let NCCL use RDMA; without them NCCL silently falls back to TCP and decode slows by roughly 40%.
- **Env knobs** in the cell are part of the recipe: `SGLANG_SM120_FLASHMLA_BACKEND=b12x` selects the b12x attention path, `SGLANG_B12X_MAX_TOKENS` must equal `--chunked-prefill-size`, and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` avoids unified-memory fragmentation OOMs on GB10.
- **Text only** — the image predates Flash Vision support, and the b12x kernels do not yet cover image prefill on SM12x; there is no DGX Spark cell for the Flash Vision variant.
- **Env knobs** in the cells are part of the recipe: `SGLANG_SM120_FLASHMLA_BACKEND=b12x` selects the b12x attention path, `SGLANG_B12X_MAX_TOKENS` must equal `--chunked-prefill-size`, and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` avoids unified-memory fragmentation OOMs on GB10.
- **NVFP4 (`nvidia/DeepSeek-V4-Flash-0731-NVFP4`)** — only the routed experts are NVFP4; attention, shared experts and the DSpark MTP layer stay in the checkpoint's native formats. On SM12x that means three extra flags: `--moe-runner-backend flashinfer_cutlass` (b12x's MoE is MXFP4-only and trtllm-gen kernels are sm100-only), `--speculative-moe-runner-backend b12x` (the DSpark draft's MTP experts are MXFP4 and run on b12x), and `--disable-shared-experts-fusion` (HashTopK rejects fused shared experts under the cutlass runner). Throughput and DSpark acceptance match the FP4 cell within noise.
- **Flash Vision** — images are served natively on the b12x recipe with the same flags as Flash Official (send `image_url` content on `/v1/chat/completions`, see [Vision](#3-5-vision-image-inputs)); text-only requests work unchanged. Expect roughly 15–20% lower text throughput than Flash Official on this checkpoint — its bundled DSpark head accepts fewer drafts (~3.2 vs ~3.9) — with text accuracy intact.
<a id="vision-note" />
@@ -232,7 +233,7 @@ The **DGX Spark** row has a single recipe: **Flash Official (0731) · FP4 · Bal
[`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) is DeepSeek's first experimental multimodal V4 checkpoint: the 0731 Flash base plus a vision encoder and aligner, served through the same `sglang serve` flow with OpenAI-style `image_url` inputs (see [Vision](#3-5-vision-image-inputs) below). Select the **Flash Vision** variant in the Deploy panel for its recipes.
- **Preview build required** — support lands via [sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253) and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image `lmsysorg/sglang:dev-dsv4-flash-vision`; for a Python environment, install SGLang from that PR's branch.
- **Preview build required** — support lands via [sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253) and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image `lmsysorg/sglang:dev-dsv4-flash-vision`; for a Python environment, install SGLang from that PR's branch. The DGX Spark Flash Vision cell is the exception: it uses the DGX Spark image `lmsysorg/sglang:dev-v4f-2dgx-v2` (see the [DGX Spark notes](#spark-note)).
- **Verified matrix** — MMMU-Pro via sgl-eval at `temperature 1.0`, `top-p 0.95`, `--reasoning-effort max`.
- **Engine auto-configuration** — the engine picks the `flashinfer_mxfp4` MoE runner and auto-disables shared-experts fusion for this checkpoint (its HashTopK routing rejects fused shared experts); don't pass `--enforce-shared-experts-fusion`.
- **Chunked prefill & radix cache stay enabled** — the scheduler keeps image spans consistent automatically: chunked-prefill truncation points are span-aligned (an image span always prefills within a single extend, overshooting the chunk budget by at most one span), and a radix-cache prefix match ending deep inside an image span is re-issued from the span start.