minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[diffusion] docs: refresh the VDN-H3 on b200 numbers in cookbook (#39244)

Browse Source 
Co-authored-by: haochengxi <xihc@berkeley.edu>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
This commit is contained in:
Haocheng Xi
2026-09-14 11:50:40 +08:00
committed by GitHub
co-authored by haochengxi Claude Opus 5 Kevin Mi
parent 96a95171c7
commit 42b5af8c62
2 changed files with 93 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx
+92 -117
View File
@@ -802,32 +802,31 @@ latency.
## 7. VDN-H3: hybrid attention, 8-step distill
[Video DeltaNet](https://openvdn.github.io/) (VDN) is a hybrid attention
architecture that approaches the speed of linear attention while maintaining
near-lossless quality. It splits the video-video attention into two
complementary branches: a sliding-window softmax branch for local frame pairs
(five consecutive latent frames form a chunk, and each chunk attends to
itself, the previous chunk, and the following chunk; the first and last
frames, text and audio stay dense) and a bidirectional linear branch, Video
Delta Attention, for long-range context.
[OpenVDN/vdn-minimax-h3](https://huggingface.co/OpenVDN/vdn-minimax-h3)
(Video DeltaNet MiniMax-H3, [openvdn.github.io](https://openvdn.github.io/))
replaces every DiT block's dense self-attention with a hybrid of two branches:
an exact, gated softmax over a chunk-aligned frame window (chunk 5, radius 1,
the first and last frames dense as anchors, text and audio dense both ways)
and a frame-wise linear-attention branch (the Video Delta rule, scanned
forward and backward over frames) that covers exactly the rest. The released
`stage-dmd-step-250` checkpoint is an 8-NFE DMD2 distill and adds a 4.3 GB
linear branch plus two small LoRA adapters on top of the untouched MiniMax-H3
backbone. The weights are the FL2VA partition, so the deployment serves `t2va`
and `fl2va` (first / last keyframes); the keyframe rows are attended densely,
like text and audio. `ref2va` was not trained and is rejected. The weights
inherit the MiniMax-H3 Community License, including its territorial exclusions.
applies it to MiniMax-H3: the released `stage-dmd-step-250` checkpoint adds a
4.3 GB linear branch and OpenVDN's own 8-step DMD2 LoRA on the unchanged H3
backbone. It is the FL2VA partition, so it serves `t2va` and `fl2va` (first,
last, or both keyframes); `ref2va` is rejected. The weights inherit the
MiniMax-H3 Community License.
Pass the repository directly to `--model-path`. A registered model overlay
materializes the base-H3 layout once: both LoRA adapters are prefused into the
transformer weights (a real 62 GB write), the linear branch is attached as an
extra shard, the Qwen3-VL conditioner is linked from `MiniMaxAI/MiniMax-H3`,
and the video VAE is re-serialized as for FastH3. Point
`SGLANG_DIFFUSION_CACHE_ROOT` at a volume with at least 90 GB free before the
first launch.
Pass the repository directly to `--model-path`. On first launch a registered
overlay prefuses both adapters into the transformer weights (a 62 GB write)
and attaches the linear branch; the Qwen3-VL conditioner and the VAEs are
hard-linked from `MiniMaxAI/MiniMax-H3`. Keep `SGLANG_DIFFUSION_CACHE_ROOT` on
the same filesystem as the Hugging Face cache with at least 90 GB free.
```bash 4×B200 hybrid window attention
```bash 8×B200
sglang serve \
--model-path OpenVDN/vdn-minimax-h3 \
--num-gpus 4 \
--num-gpus 8 \
--attention-backend hybrid_window_attn_h3 \
--performance-mode speed \
--warmup-num-frames 345 \
@@ -835,45 +834,33 @@ sglang serve \
--port 30010
```
`--warmup-num-frames` / `--warmup-resolutions` make the startup warmup run
at the clip length and canvas you will serve (here the 14.375 s paper
workload). Without them H3 warms up on a 5-second clip, and the first
forward of the first longer request pays allocator growth and kernel setup
in every DiT block (2 to 3 s on this workload, for base H3 as well).
`--num-gpus 4`, `2` or `1` gives the smaller Ulysses recipes in the benchmark
table; `sglang generate` takes the same flags. `--warmup-num-frames` /
`--warmup-resolutions` warm up at the clip length and canvas you will serve;
without them the first long request pays 2 to 3 s of allocator growth and
kernel setup.
Requests use the same asynchronous video endpoint as the base model, with
`task: "t2va"`, `conditions: []` (or `task: "fl2va"` with the keyframe
conditions of the [FL2VA request](#4-generate-video-and-audio)), and a target such as
`{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 14.375}`. Each
keyframe adds about 2,000 global rows (the conditioner's image tokens plus the
latent keyframe), dense both ways in the window softmax, so an `fl2va` forward
costs more than a `t2va` one. The
request default is `num_inference_steps: 9`: nine points on the standard
shift-12/shift-3 sigma grid, i.e. the eight distilled DiT evaluations (VDN
counts NFEs, SGLang counts grid points). Any other step count is rejected. The
paper workload is 1344×768 at 24 fps for 14.375 s: 345 frames, already 17n+5
aligned, 102 latent frames and about 104.5k packed rows.
Requests use the same video endpoint as the base model with `task: "t2va"`
and `conditions: []`, or `task: "fl2va"` with the
[keyframe conditions](#4-generate-video-and-audio), and a target such as
`{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 14.375}` (the
345-frame paper workload). The default `num_inference_steps: 9` is the only
accepted value: nine sigma grid points, eight DiT forwards. Each keyframe adds
about 2,000 dense rows, so an `fl2va` forward costs a little more than `t2va`.
`hybrid_window_attn_h3` is required for the transformer: a dense backend on
these weights would silently skip the linear branch and the softmax gates and
produce the wrong model, so it is rejected. The window softmax runs as a union
of dense FlashAttention varlen calls, exact to bf16 rounding. The
linear branch runs on fused Triton kernels (temporal conv + SiLU + L2 norm,
statistics prologue, gated RMSNorm epilogue). On Blackwell (SM100 / SM103,
and SM120 such as the RTX PRO 6000) the transformer defaults to online MXFP8
(`--quantization fp8` selects it too, `--quantization bf16` opts out); the
gates, beta, alpha and the conv stay bf16. Before SM100 `--quantization fp8`
is the per-channel fp8 path of base H3 (SM89 and SM90); SM80 has no fp8
tensor cores, so it runs the bf16 DiT (62 GB, budget for
`--layerwise-offload-components` or `--dit-layerwise-offload`). Ampere and
Ada run the window on FA3's Sm80 mainloop, which is FA2-class throughput; they
are enabled but not benchmarked. Ulysses sequence parallelism is supported;
`--model-variant`, `quality: "high"`, `--ring-degree` greater than 1,
`torch.compile`, and breakable CUDA graph execution are rejected. See
[Attention Backends](/docs/sglang-diffusion/attention_backends) for the
backend options.
`hybrid_window_attn_h3` is required: a dense backend would silently skip the
linear branch and the gates and produce the wrong model. On Blackwell
(SM100 / SM103 / SM120) the transformer defaults to online MXFP8;
`--quantization bf16` opts out. Before SM100, `--quantization fp8` is base
H3's per-channel fp8 path, and SM80 runs the bf16 DiT (62 GB; budget for
`--layerwise-offload-components`). Ampere and Ada are enabled but not
benchmarked. Ulysses sequence parallelism is supported; `--model-variant`,
`quality: "high"`, `--ring-degree` greater than 1, `torch.compile`, and
breakable CUDA graph execution are rejected. The backend's
`--attention-backend-config` keys are listed in
[Attention Backends](/docs/sglang-diffusion/attention_backends).
Measured latencies for the 4× B200 recipe are in
Measured latencies for 1, 2, 4 and 8 B200s are in
[VDN-H3 on B200](#vdn-h3-on-b200); single-card and PCIe multi-card numbers
for the RTX PRO 6000 are in [VDN-H3 on RTX PRO 6000](#vdn-h3-on-rtx-pro-6000).
@@ -1531,74 +1518,62 @@ peak memory.
### VDN-H3 on B200
A 4× B200 (SM100, 183 GB) host served [VDN-H3](#7-vdn-h3-hybrid-attention-8-step-distill)
An 8× B200 (SM100, 183 GB) host served [VDN-H3](#7-vdn-h3-hybrid-attention-8-step-distill)
on the paper workload: 1344×768 at 24 fps with audio for 14.375 s (345 frames,
102 latent frames, about 104k packed rows), `task: "t2va"`,
`num_inference_steps: 9` (8 DiT forwards), seed 1000, `hybrid_window_attn_h3`
with the decomposed window kernel, eager, and the warmup run at the served
clip shape (`--warmup-num-frames 345 --warmup-resolutions 1344x768`), so
every forward of the served request is steady state; without those flags the
first forward pays about 3 s of allocator growth and kernel setup. "Steady
`num_inference_steps: 9` (8 DiT forwards), seed 1000, `hybrid_window_attn_h3`,
eager, and the warmup run at the served clip shape
(`--warmup-num-frames 345 --warmup-resolutions 1344x768`), so every forward
of the served request is steady state; without those flags the first forward
pays 2 to 3 s of allocator growth and kernel setup. "Steady
s/NFE" is the mean of forwards 2 to 8. The OpenVDN reference rows ran the released
inference stack (`8nfe_tuned_fp8.yaml`, `infer_ulysses.py`) on the same host
and the same clip length:
and the same clip length, with its Ulysses layout (`parallel.softmax_ranks`)
swept at every GPU count rather than assumed. Every row below comes from one
campaign, one arm at a time on an otherwise idle node; a repeated arm moves by
about 3% across nodes, so treat smaller differences as noise:
| Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU |
| --- | ---: | ---: | ---: | ---: |
| OpenVDN published, FP8, 8× B200 Ulysses 5+3 ([openvdn.github.io](https://openvdn.github.io/)) | 1.40 | 11.2 s | – | – |
| OpenVDN reference, BF16 (`8nfe_tuned.yaml`), 1× B200 | 7.91 | 63.3 s | – | – |
| OpenVDN reference, FP8, 1× B200 | 6.47 | 51.7 s | – | – |
| OpenVDN reference, FP8, 4× B200 Ulysses (standard / 3+1 branch-parallel) | 2.68 / 2.61 | 21.4 / 20.9 s | – | – |
| SGLang, BF16, 1× B200, DiT layerwise offload (auto policy) | 7.86 | 64.1 s | 9.5 s | 147,608 MB |
| SGLang, FP8, 1× B200, DiT layerwise offload (auto policy) | 7.49 | 61.7 s | 9.4 s | 111,616 MB |
| SGLang, FP8, 1× B200, DiT resident (`--layerwise-offload-components text_encoder`) | 7.49 | 61.1 s | 9.3 s | 63,922 MB |
| SGLang, `--quantization bf16`, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **2.53** | **20.3 s** | 3.5 s | 97,894 MB |
| SGLang, FP8, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **2.34** | **18.7 s** | 3.4 s | 63,022 MB |
| SGLang default (online `mxfp8`), 8× B200 Ulysses8, `--performance-mode speed`, served-shape warmup | **0.93** | **7.45 s** | 1.5 s | 77,704 MB |
| SGLang, per-channel fp8 weight scales (the online `fp8` path of other models), 8× B200 Ulysses8 | 1.05 | 8.4 s | 1.5 s | 76,828 MB |
| OpenVDN published, FP8, 8× B200 Ulysses 5+3 ([openvdn.github.io](https://openvdn.github.io/)) | 1.40 | 11.2 s | — | — |
| OpenVDN reference, BF16 (`8nfe_tuned.yaml`), 1× B200 | 7.66 | 61.2 s | — | — |
| OpenVDN reference, FP8, 1× B200 | 6.25 | 50.0 s | — | — |
| OpenVDN reference, FP8, 2× B200 Ulysses (standard / 1+1 branch-parallel) | 4.87 / 5.04 | 39.0 / 40.3 s | — | — |
| OpenVDN reference, FP8, 4× B200 Ulysses (3+1 branch-parallel) | 2.59 | 20.7 s | — | — |
| OpenVDN reference, FP8, 8× B200 Ulysses (5+3 / 4+4 / standard) | 1.40 / 1.47 / 1.52 | 11.2 / 11.7 / 12.2 s | — | — |
| SGLang, `--quantization bf16`, 1× B200 | 7.45 | 58.8 s | 8.8 s | 156,298 MB |
| SGLang, `--quantization fp8` (→ online `mxfp8` on SM100+), 1× B200 | 6.03 | 47.6 s | 8.7 s | 128,418 MB |
| SGLang default (online `mxfp8`), 2× B200 Ulysses2, `--performance-mode speed`, served-shape warmup | **3.30** | **25.9 s** | 4.7 s | 106,738 MB |
| SGLang, `--quantization bf16`, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | 2.06 | 16.2 s | 2.3 s | 108,434 MB |
| SGLang default (online `mxfp8`), 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **1.67** | **13.1 s** | 2.4 s | 85,244 MB |
| SGLang default (online `mxfp8`), 8× B200 Ulysses8, `--performance-mode speed`, served-shape warmup | **0.88** | **6.9 s** | 1.5 s | 79,972 MB |
| SGLang, per-channel fp8 weight scales (the online `fp8` path of other models; on SM100+ `--quantization fp8` resolves to `mxfp8`, so this arm selects the per-channel path explicitly), 8× B200 Ulysses8 | 0.98 | 7.7 s | 1.5 s | 79,504 MB |
On 8× B200 the SGLang Ulysses8 path runs the paper workload at 0.93 s/NFE
(7.5 GPU-seconds per NFE) against the published 1.40 s/NFE of OpenVDN's 5+3
On 8× B200 the SGLang Ulysses8 path runs the paper workload at 0.88 s/NFE
(7.0 GPU-seconds per NFE) against the published 1.40 s/NFE of OpenVDN's 5+3
branch-parallel layout; the whole request (text encoding, 8 forwards, joint
decode) completes in 9.3 s after warmup. At this GPU count the step is
launch- and copy-bound in the linear branch rather than FLOP-bound: the scans
fold the 102 frames into per-chunk composites plus one chain over the chunks
with both directions per launch (the chunked window only reads states at
chunk boundaries), q/k/v and the per-head scalars travel as four async
field-major all-to-alls that land contiguous (no relayout copies for the
window K/V gathers or the branch's conv), only the 128-wide output-gate
hidden crosses the fabric (the gate's `up` runs on the head shard), the frame
mean is a reshape-sum instead of an atomic index_add, and the transformer
runs online `mxfp8` by default (e4m3 with one E8M0 scale per 32 elements into
cuBLASLt's block-scaled GEMM; the activation quant is fused into the adaLN
modulation and SwiGLU producers, so no standalone quant pass runs). The
prompt's text state joins
the frames' Cholesky batch as a virtual frame. OpenVDN's branch-parallel layout was measured slower here (1.39
s/NFE at 5+3, 1.49 at 6+2): with 11 to 12 heads per softmax rank the window
attention and its gathers grow faster than the linear ranks shrink; splitting
each rank's heads into two pipelined all-to-all groups also lost (1.10 s/NFE)
to smaller attention kernels and doubled branch launches. The profile of one
rank is 31% window FlashAttention, 24% fp8 GEMM, 12% NCCL, the rest small
kernels.
Against the published 8× B200 headline (1.40 s/NFE, 11.2
GPU-seconds per NFE), the SGLang 4× B200 FP8 run spends 9.8 GPU-seconds per
NFE, with half the GPUs and half the all-to-all fan-out. On the same host and GPU count the
SGLang path is 10-13% faster per NFE than the released stack (2.34 vs 2.61 /
2.68 s), and the 8-forward denoise is 18.7 s against their 20.9 / 21.4 s. In BF16 the single-GPU path matches the released tuned BF16 stack
(7.86 vs 7.91 s/NFE); the single-GPU FP8 gap (7.49 vs 6.47 s) is the online
FP8 GEMM path, not the hybrid attention.
decode) completes in about 8.7 s after warmup. At this GPU count the step is
launch- and copy-bound in the linear branch rather than FLOP-bound, and the
transformer runs online `mxfp8` by default (e4m3 with one E8M0 scale per 32
elements into cuBLASLt's block-scaled GEMM, with the activation quant fused
into the adaLN and SwiGLU producers).
On one B200 the auto memory policy streams the DiT layers from host memory
(the 62 GB DiT, the 66 GB conditioner and the VAEs do not all fit resident
with the 120 GB headroom the policy keeps); keeping the FP8 DiT resident with
`--layerwise-offload-components text_encoder` changes nothing measurable, so
the single-GPU rows are compute-bound. The BF16 DiT does not fit resident at
this clip length on one 183 GB card (the 104k-row activations alone take
about 100 GB). Per DiT block the hybrid attention costs about 117 ms at this
shape (window FlashAttention 46 ms, projections 24 ms, the linear branch and
its fused kernels the rest). A static-tile block-sparse Triton kernel was
measured at 226 ms against 169 ms for the decomposed FlashAttention path and
is not shipped.
Against the published 8× B200 headline (1.40 s/NFE, 11.2
GPU-seconds per NFE), the SGLang 4× B200 run spends 6.7 GPU-seconds per
NFE, with half the GPUs and half the all-to-all fan-out. At equal GPU count
SGLang is 1.48× faster at two cards (3.30 vs 4.87 s/NFE), 1.55× at four
(1.67 vs 2.59) and 1.60× at eight (0.88 vs 1.40), while on one card the two
stacks are within 4% of each other (6.03 vs 6.25 s/NFE quantized, 7.45 vs 7.66
in BF16). The advantage is therefore parallel scaling rather than per-GPU
kernel work: SGLang holds 86-91% parallel efficiency from 2 to 8 cards against
the reference stack's 56-64%, and sweeping the reference stack's own layout
does not close it (its best split is 8% better than standard Ulysses at eight
cards, 1.40 against 1.52 s/NFE).
On one B200 both single-GPU rows run with the DiT resident under the auto
memory policy (128 GB peak quantized, 156 GB in BF16, of the card's 183 GB),
so no `--layerwise-offload-components` flag is needed at this clip length and
the single-GPU rows are compute-bound.
### VDN-H3 on RTX PRO 6000
@@ -1627,8 +1602,8 @@ measured 15.7 to 17.0 s/NFE on a shared host.
| 8× RTX PRO 6000 Ulysses8 (default `mxfp8`) | 5.43 | 43.1 s | 3.5 s | 77,704 MB |
| 8× RTX PRO 6000 Ulysses8, per-channel `fp8` | 5.47 | 43.4 s | 3.5 s | 77,236 MB |
For comparison the same code on B200 measures 6.23 / 3.34 / 1.73 / 0.89 s/NFE
at 1 / 2 / 4 / 8 cards (NVLink, 87% parallel efficiency at 8).
For comparison the same code on B200 measures 6.03 / 3.30 / 1.67 / 0.88 s/NFE
at 1 / 2 / 4 / 8 cards (NVLink, 86% parallel efficiency at 8).
### H200 topology comparison