minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[diffusion] model: support VDN-H3 with a hybrid_window_attn_h3 backend (#37903)

Browse Source 
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Haocheng Xi <xihc@berkeley.edu>
Co-authored-by: Mick <mickjagger19@icloud.com>
This commit is contained in:
Kevin Mi
2026-09-12 11:36:32 +08:00
committed by GitHub
co-authored by Claude Fable 5.1 Haocheng Xi Mick
parent e91c948057
commit ff1ce11348
45 changed files with 6378 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/sglang-diffusion/attention_backends.mdx
+61
View File
@@ -89,6 +89,11 @@ For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBa
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`VIDEO_SPARSE_ATTN_H3`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Video Sparse Attention for MiniMax-H3 / FastH3 (VSA-H3). In-tree Triton block-sparse kernel (SM90 / SM100 / SM103); no external package. Configure via <code>--attention-backend-config</code>.</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`hybrid_window_attn_h3`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`HYBRID_WINDOW_ATTN_H3`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>VDN-H3 hybrid attention: chunk-aligned window softmax (exact, gated) plus the Video Delta linear branch over the window complement. The window runs as a union of FlashAttention varlen calls (FA4 on SM100 / SM103 / SM120, FA3 on SM90; on SM80 / SM86 / SM89 the same FA3 build runs its Sm80 mainloop at FA2-class speed). Configure via <code>--attention-backend-config</code>.</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`vmoba_attn`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`VMOBA_ATTN`</td>
@@ -309,6 +314,62 @@ VSA-H3 constraints:
- Ulysses sequence parallelism is supported; `--ring-degree` greater than 1,
`torch.compile`, and breakable CUDA graph execution are rejected.
**Hybrid window attention for VDN-H3 (`hybrid_window_attn_h3`)**
[VDN-H3](/cookbook/diffusion/MiniMax/MiniMax-H3#7-vdn-h3-hybrid-attention-8-step-distill)
replaces every DiT block's dense self-attention with two branches. The
softmax branch is an exact softmax over a chunk-aligned frame window (frame
`t` belongs to chunk `t // 5` and attends to chunks `c - 1 .. c + 1`; frames
0 and F-1 are dense as rows and columns; every text / audio pair stays dense)
scaled by a per-head sigmoid gate. The linear branch (a frame-wise Video Delta
rule recurrence, forward and reverse over frames) covers exactly the window's
complement and is driven by the attention module. The mask is request-static,
so the metadata is built once per request. The checkpoint's
`transformer/config.json` carries the window geometry; the backend reads it.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "24%"}} />
<col style={{width: "14%"}} />
<col style={{width: "44%"}} />
<col style={{width: "18%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`vdn_h3_dense_smoke`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Allows a dense transformer backend (`fa`) on the VDN-H3 weights for the base-H3 + LoRA equivalence smoke. The gates and the linear branch are skipped: a study, not a sample.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`false`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`vdn_max_gather_rows`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Upper bound on the gathered K/V rows per window FlashAttention call; consecutive chunk groups fill one call up to it. Splitting changes no query's kept key set, only the pass count and the gather's peak memory.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`200000`</td>
</tr>
</tbody>
</table>
Hybrid window attention constraints:
- VDN-H3 requires this backend for the transformer; a dense backend on these
weights would silently skip the linear branch and the gates, so it is
rejected unless `vdn_h3_dense_smoke` is set. The token refiner, text
encoder, and VAEs keep dense attention.
- Base MiniMax-H3 and FastH3 checkpoints have no linear branch and are
rejected by this backend.
- Ulysses sequence parallelism is supported (QK-norm + RoPE run after the
all-to-all on the head shard); `--ring-degree` greater than 1,
`torch.compile`, and breakable CUDA graph execution are rejected.
**V-MoBA (`vmoba_attn`)**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
docs/docs/sglang-diffusion/quantization.mdx
+2 -2
View File
@@ -373,7 +373,7 @@ For a pipeline whose primary DiT is named `transformer`, the shorter
MiniMax-H3 supports this path while preserving its required FP32 patch,
timestep, and output projections. See the
[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#8-feature-contracts-and-advanced-recipes)
[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#9-feature-contracts-and-advanced-recipes)
for its distributed serving recipe.
### MXFP4 Online Quantization
@@ -432,7 +432,7 @@ projections take that path.
<Warning>
`kitchen_int8` is approximate and is not a consistency ground-truth mode.
The BF16 path is unchanged when `comfy-kitchen` is not installed. See the
[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#8-feature-contracts-and-advanced-recipes)
[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#9-feature-contracts-and-advanced-recipes)
for the 24 GB offload recipe, including why `vae` must stay out of
`--layerwise-offload-components`.
</Warning>