minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[diffusion] docs: refresh the VDN-H3 on b200 numbers in cookbook (#39244)

Browse Source 
Co-authored-by: haochengxi <xihc@berkeley.edu>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
This commit is contained in:
Haocheng Xi
2026-09-14 11:50:40 +08:00
committed by GitHub
co-authored by haochengxi Claude Opus 5 Kevin Mi
parent 96a95171c7
commit 42b5af8c62
2 changed files with 93 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx
+92 -117
View File
@@ -802,32 +802,31 @@ latency.
## 7. VDN-H3: hybrid attention, 8-step distill ## 7. VDN-H3: hybrid attention, 8-step distill
[Video DeltaNet](https://openvdn.github.io/) (VDN) is a hybrid attention
architecture that approaches the speed of linear attention while maintaining
near-lossless quality. It splits the video-video attention into two
complementary branches: a sliding-window softmax branch for local frame pairs
(five consecutive latent frames form a chunk, and each chunk attends to
itself, the previous chunk, and the following chunk; the first and last
frames, text and audio stay dense) and a bidirectional linear branch, Video
Delta Attention, for long-range context.
[OpenVDN/vdn-minimax-h3](https://huggingface.co/OpenVDN/vdn-minimax-h3) [OpenVDN/vdn-minimax-h3](https://huggingface.co/OpenVDN/vdn-minimax-h3)
(Video DeltaNet MiniMax-H3, [openvdn.github.io](https://openvdn.github.io/)) applies it to MiniMax-H3: the released `stage-dmd-step-250` checkpoint adds a
replaces every DiT block's dense self-attention with a hybrid of two branches: 4.3 GB linear branch and OpenVDN's own 8-step DMD2 LoRA on the unchanged H3
an exact, gated softmax over a chunk-aligned frame window (chunk 5, radius 1, backbone. It is the FL2VA partition, so it serves `t2va` and `fl2va` (first,
the first and last frames dense as anchors, text and audio dense both ways) last, or both keyframes); `ref2va` is rejected. The weights inherit the
and a frame-wise linear-attention branch (the Video Delta rule, scanned MiniMax-H3 Community License.
forward and backward over frames) that covers exactly the rest. The released
`stage-dmd-step-250` checkpoint is an 8-NFE DMD2 distill and adds a 4.3 GB
linear branch plus two small LoRA adapters on top of the untouched MiniMax-H3
backbone. The weights are the FL2VA partition, so the deployment serves `t2va`
and `fl2va` (first / last keyframes); the keyframe rows are attended densely,
like text and audio. `ref2va` was not trained and is rejected. The weights
inherit the MiniMax-H3 Community License, including its territorial exclusions.
Pass the repository directly to `--model-path`. A registered model overlay Pass the repository directly to `--model-path`. On first launch a registered
materializes the base-H3 layout once: both LoRA adapters are prefused into the overlay prefuses both adapters into the transformer weights (a 62 GB write)
transformer weights (a real 62 GB write), the linear branch is attached as an and attaches the linear branch; the Qwen3-VL conditioner and the VAEs are
extra shard, the Qwen3-VL conditioner is linked from `MiniMaxAI/MiniMax-H3`, hard-linked from `MiniMaxAI/MiniMax-H3`. Keep `SGLANG_DIFFUSION_CACHE_ROOT` on
and the video VAE is re-serialized as for FastH3. Point the same filesystem as the Hugging Face cache with at least 90 GB free.
`SGLANG_DIFFUSION_CACHE_ROOT` at a volume with at least 90 GB free before the
first launch.
```bash 4×B200 hybrid window attention ```bash 8×B200
sglang serve \ sglang serve \
--model-path OpenVDN/vdn-minimax-h3 \ --model-path OpenVDN/vdn-minimax-h3 \
--num-gpus 4 \ --num-gpus 8 \
--attention-backend hybrid_window_attn_h3 \ --attention-backend hybrid_window_attn_h3 \
--performance-mode speed \ --performance-mode speed \
--warmup-num-frames 345 \ --warmup-num-frames 345 \
@@ -835,45 +834,33 @@ sglang serve \
--port 30010 --port 30010
``` ```
`--warmup-num-frames` / `--warmup-resolutions` make the startup warmup run `--num-gpus 4`, `2` or `1` gives the smaller Ulysses recipes in the benchmark
at the clip length and canvas you will serve (here the 14.375 s paper table; `sglang generate` takes the same flags. `--warmup-num-frames` /
workload). Without them H3 warms up on a 5-second clip, and the first `--warmup-resolutions` warm up at the clip length and canvas you will serve;
forward of the first longer request pays allocator growth and kernel setup without them the first long request pays 2 to 3 s of allocator growth and
in every DiT block (2 to 3 s on this workload, for base H3 as well). kernel setup.
Requests use the same asynchronous video endpoint as the base model, with Requests use the same video endpoint as the base model with `task: "t2va"`
`task: "t2va"`, `conditions: []` (or `task: "fl2va"` with the keyframe and `conditions: []`, or `task: "fl2va"` with the
conditions of the [FL2VA request](#4-generate-video-and-audio)), and a target such as [keyframe conditions](#4-generate-video-and-audio), and a target such as
`{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 14.375}`. Each `{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 14.375}` (the
keyframe adds about 2,000 global rows (the conditioner's image tokens plus the 345-frame paper workload). The default `num_inference_steps: 9` is the only
latent keyframe), dense both ways in the window softmax, so an `fl2va` forward accepted value: nine sigma grid points, eight DiT forwards. Each keyframe adds
costs more than a `t2va` one. The about 2,000 dense rows, so an `fl2va` forward costs a little more than `t2va`.
request default is `num_inference_steps: 9`: nine points on the standard
shift-12/shift-3 sigma grid, i.e. the eight distilled DiT evaluations (VDN
counts NFEs, SGLang counts grid points). Any other step count is rejected. The
paper workload is 1344×768 at 24 fps for 14.375 s: 345 frames, already 17n+5
aligned, 102 latent frames and about 104.5k packed rows.
`hybrid_window_attn_h3` is required for the transformer: a dense backend on `hybrid_window_attn_h3` is required: a dense backend would silently skip the
these weights would silently skip the linear branch and the softmax gates and linear branch and the gates and produce the wrong model. On Blackwell
produce the wrong model, so it is rejected. The window softmax runs as a union (SM100 / SM103 / SM120) the transformer defaults to online MXFP8;
of dense FlashAttention varlen calls, exact to bf16 rounding. The `--quantization bf16` opts out. Before SM100, `--quantization fp8` is base
linear branch runs on fused Triton kernels (temporal conv + SiLU + L2 norm, H3's per-channel fp8 path, and SM80 runs the bf16 DiT (62 GB; budget for
statistics prologue, gated RMSNorm epilogue). On Blackwell (SM100 / SM103, `--layerwise-offload-components`). Ampere and Ada are enabled but not
and SM120 such as the RTX PRO 6000) the transformer defaults to online MXFP8 benchmarked. Ulysses sequence parallelism is supported; `--model-variant`,
(`--quantization fp8` selects it too, `--quantization bf16` opts out); the `quality: "high"`, `--ring-degree` greater than 1, `torch.compile`, and
gates, beta, alpha and the conv stay bf16. Before SM100 `--quantization fp8` breakable CUDA graph execution are rejected. The backend's
is the per-channel fp8 path of base H3 (SM89 and SM90); SM80 has no fp8 `--attention-backend-config` keys are listed in
tensor cores, so it runs the bf16 DiT (62 GB, budget for [Attention Backends](/docs/sglang-diffusion/attention_backends).
`--layerwise-offload-components` or `--dit-layerwise-offload`). Ampere and
Ada run the window on FA3's Sm80 mainloop, which is FA2-class throughput; they
are enabled but not benchmarked. Ulysses sequence parallelism is supported;
`--model-variant`, `quality: "high"`, `--ring-degree` greater than 1,
`torch.compile`, and breakable CUDA graph execution are rejected. See
[Attention Backends](/docs/sglang-diffusion/attention_backends) for the
backend options.
Measured latencies for the 4× B200 recipe are in Measured latencies for 1, 2, 4 and 8 B200s are in
[VDN-H3 on B200](#vdn-h3-on-b200); single-card and PCIe multi-card numbers [VDN-H3 on B200](#vdn-h3-on-b200); single-card and PCIe multi-card numbers
for the RTX PRO 6000 are in [VDN-H3 on RTX PRO 6000](#vdn-h3-on-rtx-pro-6000). for the RTX PRO 6000 are in [VDN-H3 on RTX PRO 6000](#vdn-h3-on-rtx-pro-6000).
@@ -1531,74 +1518,62 @@ peak memory.
### VDN-H3 on B200 ### VDN-H3 on B200
A 4× B200 (SM100, 183 GB) host served [VDN-H3](#7-vdn-h3-hybrid-attention-8-step-distill) An 8× B200 (SM100, 183 GB) host served [VDN-H3](#7-vdn-h3-hybrid-attention-8-step-distill)
on the paper workload: 1344×768 at 24 fps with audio for 14.375 s (345 frames, on the paper workload: 1344×768 at 24 fps with audio for 14.375 s (345 frames,
102 latent frames, about 104k packed rows), `task: "t2va"`, 102 latent frames, about 104k packed rows), `task: "t2va"`,
`num_inference_steps: 9` (8 DiT forwards), seed 1000, `hybrid_window_attn_h3` `num_inference_steps: 9` (8 DiT forwards), seed 1000, `hybrid_window_attn_h3`,
with the decomposed window kernel, eager, and the warmup run at the served eager, and the warmup run at the served clip shape
clip shape (`--warmup-num-frames 345 --warmup-resolutions 1344x768`), so (`--warmup-num-frames 345 --warmup-resolutions 1344x768`), so every forward
every forward of the served request is steady state; without those flags the of the served request is steady state; without those flags the first forward
first forward pays about 3 s of allocator growth and kernel setup. "Steady pays 2 to 3 s of allocator growth and kernel setup. "Steady
s/NFE" is the mean of forwards 2 to 8. The OpenVDN reference rows ran the released s/NFE" is the mean of forwards 2 to 8. The OpenVDN reference rows ran the released
inference stack (`8nfe_tuned_fp8.yaml`, `infer_ulysses.py`) on the same host inference stack (`8nfe_tuned_fp8.yaml`, `infer_ulysses.py`) on the same host
and the same clip length: and the same clip length, with its Ulysses layout (`parallel.softmax_ranks`)
swept at every GPU count rather than assumed. Every row below comes from one
campaign, one arm at a time on an otherwise idle node; a repeated arm moves by
about 3% across nodes, so treat smaller differences as noise:
| Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU | | Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU |
| --- | ---: | ---: | ---: | ---: | | --- | ---: | ---: | ---: | ---: |
| OpenVDN published, FP8, 8× B200 Ulysses 5+3 ([openvdn.github.io](https://openvdn.github.io/)) | 1.40 | 11.2 s | – | – | | OpenVDN published, FP8, 8× B200 Ulysses 5+3 ([openvdn.github.io](https://openvdn.github.io/)) | 1.40 | 11.2 s | — | — |
| OpenVDN reference, BF16 (`8nfe_tuned.yaml`), 1× B200 | 7.91 | 63.3 s | – | – | | OpenVDN reference, BF16 (`8nfe_tuned.yaml`), 1× B200 | 7.66 | 61.2 s | — | — |
| OpenVDN reference, FP8, 1× B200 | 6.47 | 51.7 s | – | – | | OpenVDN reference, FP8, 1× B200 | 6.25 | 50.0 s | — | — |
| OpenVDN reference, FP8, 4× B200 Ulysses (standard / 3+1 branch-parallel) | 2.68 / 2.61 | 21.4 / 20.9 s | – | – | | OpenVDN reference, FP8, 2× B200 Ulysses (standard / 1+1 branch-parallel) | 4.87 / 5.04 | 39.0 / 40.3 s | — | — |
| SGLang, BF16, 1× B200, DiT layerwise offload (auto policy) | 7.86 | 64.1 s | 9.5 s | 147,608 MB | | OpenVDN reference, FP8, 4× B200 Ulysses (3+1 branch-parallel) | 2.59 | 20.7 s | — | — |
| SGLang, FP8, 1× B200, DiT layerwise offload (auto policy) | 7.49 | 61.7 s | 9.4 s | 111,616 MB | | OpenVDN reference, FP8, 8× B200 Ulysses (5+3 / 4+4 / standard) | 1.40 / 1.47 / 1.52 | 11.2 / 11.7 / 12.2 s | — | — |
| SGLang, FP8, 1× B200, DiT resident (`--layerwise-offload-components text_encoder`) | 7.49 | 61.1 s | 9.3 s | 63,922 MB | | SGLang, `--quantization bf16`, 1× B200 | 7.45 | 58.8 s | 8.8 s | 156,298 MB |
| SGLang, `--quantization bf16`, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **2.53** | **20.3 s** | 3.5 s | 97,894 MB | | SGLang, `--quantization fp8` (→ online `mxfp8` on SM100+), 1× B200 | 6.03 | 47.6 s | 8.7 s | 128,418 MB |
| SGLang, FP8, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **2.34** | **18.7 s** | 3.4 s | 63,022 MB | | SGLang default (online `mxfp8`), 2× B200 Ulysses2, `--performance-mode speed`, served-shape warmup | **3.30** | **25.9 s** | 4.7 s | 106,738 MB |
| SGLang default (online `mxfp8`), 8× B200 Ulysses8, `--performance-mode speed`, served-shape warmup | **0.93** | **7.45 s** | 1.5 s | 77,704 MB | | SGLang, `--quantization bf16`, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | 2.06 | 16.2 s | 2.3 s | 108,434 MB |
| SGLang, per-channel fp8 weight scales (the online `fp8` path of other models), 8× B200 Ulysses8 | 1.05 | 8.4 s | 1.5 s | 76,828 MB | | SGLang default (online `mxfp8`), 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **1.67** | **13.1 s** | 2.4 s | 85,244 MB |
| SGLang default (online `mxfp8`), 8× B200 Ulysses8, `--performance-mode speed`, served-shape warmup | **0.88** | **6.9 s** | 1.5 s | 79,972 MB |
| SGLang, per-channel fp8 weight scales (the online `fp8` path of other models; on SM100+ `--quantization fp8` resolves to `mxfp8`, so this arm selects the per-channel path explicitly), 8× B200 Ulysses8 | 0.98 | 7.7 s | 1.5 s | 79,504 MB |
On 8× B200 the SGLang Ulysses8 path runs the paper workload at 0.93 s/NFE On 8× B200 the SGLang Ulysses8 path runs the paper workload at 0.88 s/NFE
(7.5 GPU-seconds per NFE) against the published 1.40 s/NFE of OpenVDN's 5+3 (7.0 GPU-seconds per NFE) against the published 1.40 s/NFE of OpenVDN's 5+3
branch-parallel layout; the whole request (text encoding, 8 forwards, joint branch-parallel layout; the whole request (text encoding, 8 forwards, joint
decode) completes in 9.3 s after warmup. At this GPU count the step is decode) completes in about 8.7 s after warmup. At this GPU count the step is
launch- and copy-bound in the linear branch rather than FLOP-bound: the scans launch- and copy-bound in the linear branch rather than FLOP-bound, and the
fold the 102 frames into per-chunk composites plus one chain over the chunks transformer runs online `mxfp8` by default (e4m3 with one E8M0 scale per 32
with both directions per launch (the chunked window only reads states at elements into cuBLASLt's block-scaled GEMM, with the activation quant fused
chunk boundaries), q/k/v and the per-head scalars travel as four async into the adaLN and SwiGLU producers).
field-major all-to-alls that land contiguous (no relayout copies for the
window K/V gathers or the branch's conv), only the 128-wide output-gate
hidden crosses the fabric (the gate's `up` runs on the head shard), the frame
mean is a reshape-sum instead of an atomic index_add, and the transformer
runs online `mxfp8` by default (e4m3 with one E8M0 scale per 32 elements into
cuBLASLt's block-scaled GEMM; the activation quant is fused into the adaLN
modulation and SwiGLU producers, so no standalone quant pass runs). The
prompt's text state joins
the frames' Cholesky batch as a virtual frame. OpenVDN's branch-parallel layout was measured slower here (1.39
s/NFE at 5+3, 1.49 at 6+2): with 11 to 12 heads per softmax rank the window
attention and its gathers grow faster than the linear ranks shrink; splitting
each rank's heads into two pipelined all-to-all groups also lost (1.10 s/NFE)
to smaller attention kernels and doubled branch launches. The profile of one
rank is 31% window FlashAttention, 24% fp8 GEMM, 12% NCCL, the rest small
kernels.
Against the published 8× B200 headline (1.40 s/NFE, 11.2
GPU-seconds per NFE), the SGLang 4× B200 FP8 run spends 9.8 GPU-seconds per
NFE, with half the GPUs and half the all-to-all fan-out. On the same host and GPU count the
SGLang path is 10-13% faster per NFE than the released stack (2.34 vs 2.61 /
2.68 s), and the 8-forward denoise is 18.7 s against their 20.9 / 21.4 s. In BF16 the single-GPU path matches the released tuned BF16 stack
(7.86 vs 7.91 s/NFE); the single-GPU FP8 gap (7.49 vs 6.47 s) is the online
FP8 GEMM path, not the hybrid attention.
On one B200 the auto memory policy streams the DiT layers from host memory Against the published 8× B200 headline (1.40 s/NFE, 11.2
(the 62 GB DiT, the 66 GB conditioner and the VAEs do not all fit resident GPU-seconds per NFE), the SGLang 4× B200 run spends 6.7 GPU-seconds per
with the 120 GB headroom the policy keeps); keeping the FP8 DiT resident with NFE, with half the GPUs and half the all-to-all fan-out. At equal GPU count
`--layerwise-offload-components text_encoder` changes nothing measurable, so SGLang is 1.48× faster at two cards (3.30 vs 4.87 s/NFE), 1.55× at four
the single-GPU rows are compute-bound. The BF16 DiT does not fit resident at (1.67 vs 2.59) and 1.60× at eight (0.88 vs 1.40), while on one card the two
this clip length on one 183 GB card (the 104k-row activations alone take stacks are within 4% of each other (6.03 vs 6.25 s/NFE quantized, 7.45 vs 7.66
about 100 GB). Per DiT block the hybrid attention costs about 117 ms at this in BF16). The advantage is therefore parallel scaling rather than per-GPU
shape (window FlashAttention 46 ms, projections 24 ms, the linear branch and kernel work: SGLang holds 86-91% parallel efficiency from 2 to 8 cards against
its fused kernels the rest). A static-tile block-sparse Triton kernel was the reference stack's 56-64%, and sweeping the reference stack's own layout
measured at 226 ms against 169 ms for the decomposed FlashAttention path and does not close it (its best split is 8% better than standard Ulysses at eight
is not shipped. cards, 1.40 against 1.52 s/NFE).
On one B200 both single-GPU rows run with the DiT resident under the auto
memory policy (128 GB peak quantized, 156 GB in BF16, of the card's 183 GB),
so no `--layerwise-offload-components` flag is needed at this clip length and
the single-GPU rows are compute-bound.
### VDN-H3 on RTX PRO 6000 ### VDN-H3 on RTX PRO 6000
@@ -1627,8 +1602,8 @@ measured 15.7 to 17.0 s/NFE on a shared host.
| 8× RTX PRO 6000 Ulysses8 (default `mxfp8`) | 5.43 | 43.1 s | 3.5 s | 77,704 MB | | 8× RTX PRO 6000 Ulysses8 (default `mxfp8`) | 5.43 | 43.1 s | 3.5 s | 77,704 MB |
| 8× RTX PRO 6000 Ulysses8, per-channel `fp8` | 5.47 | 43.4 s | 3.5 s | 77,236 MB | | 8× RTX PRO 6000 Ulysses8, per-channel `fp8` | 5.47 | 43.4 s | 3.5 s | 77,236 MB |
For comparison the same code on B200 measures 6.23 / 3.34 / 1.73 / 0.89 s/NFE For comparison the same code on B200 measures 6.03 / 3.30 / 1.67 / 0.88 s/NFE
at 1 / 2 / 4 / 8 cards (NVLink, 87% parallel efficiency at 8). at 1 / 2 / 4 / 8 cards (NVLink, 86% parallel efficiency at 8).
### H200 topology comparison ### H200 topology comparison
python/sglang/multimodal_gen/runtime/utils/model_overlay.py
+1 -1
View File
@@ -48,7 +48,7 @@ BUILTIN_MODEL_OVERLAY_REGISTRY: dict[str, dict[str, Any]] = {
}, },
"OpenVDN/vdn-minimax-h3": { "OpenVDN/vdn-minimax-h3": {
"overlay_repo_id": "kevin-mi/VDN-H3-overlay", "overlay_repo_id": "kevin-mi/VDN-H3-overlay",
"overlay_revision": "0ad315a05b914c4003af4d26152d288c2506a609", "overlay_revision": "7de18275dddfe59da36a234e222bcdd274963bc3",
}, },
} }