minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

[diffusion] docs: give the RTX 5090 its own H3 recipe, measured on a physical desktop (#39373)

Browse Source 
Co-authored-by: Mick Qian <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Mick
2026-09-15 09:15:13 +08:00
committed by GitHub
co-authored by Mick Qian Claude Opus 5
parent fc4193a63c
commit a25f213bc4
2 changed files with 134 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx
+90 -14
View File
@@ -1369,7 +1369,7 @@ the configurations with collected measurements:
| H100 | 4× TP2 + Ulysses2 resident | 4× TP4 + Ulysses1; 4× FSDP + Ulysses4 |
| Ascend NPU | 8 NPUs, TP2 + SP4, Laser Attention | 4 NPUs, TP2 + SP2, Laser Attention |
| MI300X / MI355X | 8× Ulysses8 resident | 1×, 2×, and 4× scaling runs |
| RTX 5090 | 2× TP2 + layerwise offload | — |
| RTX 5090 | 1× layerwise offload, measured on a physical desktop | 2× TP2 + layerwise offload |
| RTX 4090 24 GB | 1× layerwise offload + `kitchen_int8` | Approximate attention backends are opt-in |
### GB300 single-host
@@ -1706,9 +1706,11 @@ Ulysses2 was the fastest; TP4 used the least memory:
| FSDP + Ulysses4 | 13.36 s | 57.01 GB |
| TP4 + Ulysses1 | 13.86 s | 49.80 GB |
### RTX 5090 capacity run
### RTX 5090 capacity run (two cards)
The verified two-card RTX 5090 host used TP2 with layerwise offload. The full
The single-card recipe is a different one — see *RTX 5090 single card, physical
desktop* below; the twenty resident layers here are half a layer per GPU under
TP2 and do not transfer to one card. The verified two-card RTX 5090 host used TP2 with layerwise offload. The full
50-step, 1344×768, 5-second request completed in 559.67 seconds: 525.05
seconds of denoising and 33.61 seconds of decoding, with a 26.3 GiB sampled
peak per GPU.
@@ -1885,16 +1887,33 @@ Under that cap, Recipe A wins the whole request at every host size:
Same GPU, same load window, unpruned bf16 checkpoints, outputs verified. All
figures are anchored at 480P — activations grow with the pixel count, so at
768P drop the resident DiT layers to 0 first, then `video_vae` to 24 if the
decode still collides. And the host convention holds the weights in page
cache; a physical 32 GB machine re-reads them from disk each step, so the
page cache cannot hold the per-step weight sweep, so every step re-reads it
from disk and the drive becomes the denoise clock: a real desktop 4090 with a
990 Pro measured 38 s/step, reading 52.9 GB per step (faulted sequentially, so
almost none of it shows in majflt — measure `read_bytes`, not major faults).
Two things cut that read directly: resident DiT layers (~1 GB/step each — on a
physically small host raise them as far as VRAM allows, the opposite of the
capped-host guidance above), and more RAM (64 GB caches the sweep and returns
to the quoted times). The
decode still collides.
One convention behind that table is the one a home machine does not share: the
host sizes above are `psutil` patches on a 2 TB lab machine, so the kernel kept
the whole checkpoint in page cache and nothing was read from disk. A physical
host of that size re-reads the streamed weights every step, and what that costs
is computable — `bytes per step = (50 - resident DiT layers - layers the host
pin budget covered) x 1.23 GiB`. The courier overlaps that read with compute, so
the drive stops being the denoise clock once its sequential rate exceeds those
bytes divided by the card's step time. A 32 GB host covers roughly 13 layers,
which puts a 24 GB card at ~38 GiB/step and a 12/16 GB card at ~46 GiB/step: a
PCIe 4.0 NVMe (6–7 GiB/s) clears both, a PCIe 3.0 drive (~3.2 GiB/s) becomes the
clock, and a SATA SSD is not usable for this model. Resident DiT layers remove
1.23 GiB/step each, but only while the streamed set is larger than the pin
budget — once the read fits under the step's compute time, more of them buy
nothing and only cost VRAM.
An earlier revision of this page recorded 38 s/step on a physical 4090 with a
990 Pro, reading 52.9 GB per step — about 1.4 GB/s, and faulted sequentially, so
almost none of it showed in `majflt` (measure `read_bytes`, not major faults).
That rate was the old path, not the drive: the pinned-store pool rounded every
block up to a power of two, so the pin budget over-committed and the machine had
to run with pinning off, and the layers that stayed mapped faulted in 4 KiB at a
time. Exact-size pinning and O_DIRECT reads through the courier removed both,
and the physical 5090 run below shows the drive fully hidden behind compute. A
4090 on a 32 GB host is derived to return to its compute wall (~8.5 s/step) on a
PCIe 4.0 drive; that machine has not been re-measured. The
VRAM axis holds too: capped at 16 GiB the same recipe wins ~250 vs 292–301 s,
and at 24 GiB (with `--dit-layerwise-resident-layers 6` — measured at a
22 GiB cap so a desktop's own allocations fit; a headless card can raise it
@@ -1911,7 +1930,9 @@ Stage by stage under the cap: text encoding is even (both stream the same
48 GB Qwen3VL), the denoise leads at 32–48 GB hosts and sits within
run-to-run variance of ComfyUI at 64 GB (162 vs 159 s), and the decode leads
everywhere. Two ComfyUI notes that still matter: `--fast-disk` measured no
faster than its default here, and stacking
faster than its default on this lab host — though on a physical 60 GB desktop
its default is OOM-killed while loading (55.6 GB of anonymous memory) and
`--fast-disk` is what makes it run at all — and stacking
`--novram --cache-none --disable-pinned-memory` made things strictly worse
(69.1 GiB anonymous, 750 s requests) — the adaptive default is the right
configuration on a small host.
@@ -1922,6 +1943,61 @@ i.e. an int8 DiT and an NVFP4 text encoder, and its pruned bf16 file is 40.2 GB
against the unpruned 66.3 GB. Those are different weights, so it is not a
like-for-like comparison with the recipes above.
### RTX 5090 single card, physical desktop
Every consumer figure above this point was measured with at least one axis
simulated: a hard allocator cap standing in for VRAM, a `psutil` patch standing
in for host RAM, and a lab machine whose page cache held the whole checkpoint.
This run had none of that — an RTX 5090 (32 GB) in a Ryzen 9 9950X desktop with
60 GB of RAM, an 8 GB swap file and a Samsung 9100 PRO on PCIe 5.0, on Ubuntu
24.04. The workload matches the comparison above: FL2VA T2VA, 864×480, 124
frames, 20 steps, cfg 1.0, euler_ancestral, sigma shift 12.0/3.0, same seeds and
keyframes on both engines, unpruned bf16 weights on both sides.
```bash 32 GB VRAM + 60 GB host, lossless
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \
--performance-mode memory \
--layerwise-offload-components dit,text_encoder,vae \
--dit-layerwise-resident-layers 14 \
--layerwise-resident-layers video_vae=36 \
--warmup-resolutions 864x480 --warmup-num-frames 124
```
| | text encoding | denoise step | decode | request |
| --- | ---: | ---: | ---: | ---: |
| sglang | **4.2 s** | 5.14–5.17 s | **6.4 s** | **112.2 / 112.1 s** |
| ComfyUI `--fast-disk --cache-none` | 11.3 s | 5.17–5.30 s | 7.2 s | 145.9 / 145.6 / 140.9 s |
Three things a capped run cannot settle:
**The denoise is at the card's compute wall, not the drive's.** The two engines
are within noise of each other per step, and prefetch depth 2 and cuDNN SDPA
each measured as no change. The 51.3 GiB the DiT reads from the drive per
request is fully overlapped — across a whole request the compute thread waited
0.9 s for the courier.
**Host RAM decides how much traffic there is to hide.** At 60 GB the pin budget
covers 34 of the 50 DiT layers, so only about two layers per step come off the
disk. The 46 GiB text encoder is never pinned — streamed layers take the budget
first — and is read whole, once per request, at the drive's sequential rate:
that is the 4.2 s stage, against 12.1–12.9 s when the same bytes go through the
page cache.
**Neither engine's default command survives this machine.** ComfyUI's is
OOM-killed during load (55.6 GB of anonymous memory) and needs `--fast-disk`;
sglang's was too, until pinned stores were allocated at their exact size instead
of the allocator's next power of two. Even now the automatic placement keeps no
resident DiT layers and streams the VAE on every decode, so the flags above are
what reach the numbers in the table.
Two caveats to carry into your own machine. The run finished with its 8 GB swap
file full, because the pin budget takes 95% of the memory available after
loading and there is no smaller setting yet — on a machine that is also your
desktop, expect that pressure and give the card more resident layers, since a
resident layer is one the host no longer has to pin. And these are 480P figures:
at 768P drop the resident DiT layers first, then `video_vae` to 24.
### RTX 4090 24 GB single-GPU run
One RTX 4090 D 24 GB completed the 1344×768, 107-frame, 20-NFE T2VA
docs/src/snippets/configs/MiniMaxAI/minimax-h3.jsx
+44 -9
View File
@@ -19,6 +19,10 @@ export const config = (() => {
const CONSUMER_12G = ["rtx4070", "rtx5070", "rtx3060"];
const CONSUMER_16G = ["rtx4080", "rtx5080", "rtx5070ti", "rtx4060ti"];
const CONSUMER_24G = ["rtx4090", "rtx3090"];
// The only tier measured on a physical desktop rather than under an allocator
// cap: an RTX 5090 with 60 GB of host RAM and a PCIe 5.0 NVMe, where the VRAM,
// the host RAM and the drive are all real at once.
const CONSUMER_32G = ["rtx5090"];
// Workstation cards a home builder can actually buy. No hard-cap anchor was
// measured for these sizes (the lab card is 24 GB and caps only shrink), so
// their recipes are derived from the tier logic, not verified runs.
@@ -34,13 +38,24 @@ const CONSUMER_SINGLE = [
...CONSUMER_12G,
...CONSUMER_16G,
...CONSUMER_24G,
...CONSUMER_32G,
...WORKSTATION_48G,
...WORKSTATION_96G,
...UNIFIED_128G,
];
const CONSUMER_VRAM_16_PLUS = [...CONSUMER_16G, ...CONSUMER_24G];
const CONSUMER_VRAM_16_PLUS = [...CONSUMER_16G, ...CONSUMER_24G, ...CONSUMER_32G];
const CONSUMER_AMPERE = ["rtx3060", "rtx3090"];
// The consumer recipes below are single-card. The 5090 is the one consumer
// card with a verified two-card recipe (TP2, twenty resident layers -- half a
// layer per GPU), so a two-card selection keeps the generic offload path.
function consumerSingleCard(s) {
return (
CONSUMER_SINGLE.includes(s.hw)
&& !(CONSUMER_32G.includes(s.hw) && Number(s.gpus_per_node) > 1)
);
}
function consumerFlags(s) {
if (UNIFIED_128G.includes(s.hw)) return unified128Flags();
if (WORKSTATION_96G.includes(s.hw)) return workstation96Flags();
@@ -67,6 +82,13 @@ function consumerFlags(s) {
if (CONSUMER_24G.includes(s.hw) && s.host_ram === "ram32") {
flags.push("--dit-layerwise-resident-layers 6");
}
// Fourteen layers (~17 GiB) leave the decode its room on a 32 GB card and
// shrink the streamed set the host has to pin. Measured at a 60 GB host;
// a smaller host does not change what fits on the card, so the count holds
// there too and matters more, since fewer streamed layers get pinned.
if (CONSUMER_32G.includes(s.hw) && s.host_ram !== "ram96") {
flags.push("--dit-layerwise-resident-layers 14");
}
if (WORKSTATION_48G.includes(s.hw)) {
flags.push("--dit-layerwise-resident-layers 40");
}
@@ -97,14 +119,23 @@ function consumerHints(s) {
const midHost = s.host_ram === "ram64";
if (bigHost) {
if (CONSUMER_VRAM_16_PLUS.includes(s.hw)) {
hints.push("verified end to end: ~6 s per denoise step, 13 s decode");
hints.push(CONSUMER_32G.includes(s.hw)
? "with the DiT pinned the denoise runs at this card's compute wall, measured at 5.14-5.17 s per step on a physical 5090; the decode holds all 36 blocks in their fp16 decode dtype and takes ~6.4 s"
: "verified end to end: ~6 s per denoise step, 13 s decode");
hints.push("fewer resident layers than the 32 GB rows is not a typo: with the DiT pinned in a big host, streamed layers arrive at pinned-copy speed and GPU residency buys little; on a 32 GB host the stream is the bottleneck residency cuts");
} else {
hints.push("~6 s per step once the host pins the DiT; the decode holds all 36 blocks in their fp16 decode dtype and takes ~10 s");
}
return hints;
}
if (CONSUMER_24G.includes(s.hw)) {
if (CONSUMER_32G.includes(s.hw)) {
if (midHost) {
hints.push("measured end to end on a physical desktop (RTX 5090, 60 GB host, PCIe 5.0 NVMe -- VRAM, host RAM and drive all real): 112.2 / 112.1 s per request at 864x480 / 124 frames / 20 steps, against ComfyUI's 140.9-145.9 s on the same weights and sampler settings. Text encoding 4.2 s, denoise 98.0-98.5 s (5.14-5.17 s/step), decode 6.4 s");
hints.push("at this host size the pin budget covers 34 of the 50 DiT layers, so only ~2 layers per step and the 46 GiB text encoder come off the drive -- 96 GiB per request, read with O_DIRECT and fully hidden behind compute (the compute thread waited 0.9 s per request)");
} else {
hints.push("derived, not yet measured at this host size: the card holds the same fourteen resident layers, but the pin budget covers only ~13 of the 36 streamed layers, so ~28 GiB per step comes off the drive. That stays hidden behind the 5.14 s/step compute wall on a drive delivering ~5.5 GiB/s or better (a PCIe 4.0 NVMe); a PCIe 3.0 drive becomes the clock at ~8.8 s/step");
}
} else if (CONSUMER_24G.includes(s.hw)) {
hints.push("measured at 32 GB host under a 22 GiB cap (desktop headroom): ~8.5 s per denoise step with six resident layers, ~9.6 s decode -- ahead of ComfyUI (249-260 s at the 24 GiB cap); a headless card can raise to ten layers for under 1% more");
} else if (CONSUMER_16G.includes(s.hw)) {
hints.push("measured at 32 GB host: ~11.9 s per denoise step, ~11 s decode, ~250 s per request -- ahead of ComfyUI (292-301 s) under the same hard 16 GiB cap");
@@ -132,11 +163,15 @@ function consumerHints(s) {
if (midHost) {
hints.push("measured on a 12 GB card at a 48 GB host: ~9.6 s/step, ~218 s per request (ComfyUI 246-267 s); at 64 GB: ~8.1 s/step, ~180 s (ComfyUI 194-195 s); larger cards land at or below these");
} else {
hints.push("a 32 GB host cannot cache the 108 GB checkpoint: NVMe is required, and real runs land above the quoted step time");
hints.push("a 32 GB host cannot cache the 108 GB checkpoint, so the drive is in the loop on every step -- see the rate it has to clear, below. A physical desktop that clears it reaches the quoted step times rather than falling short of them");
}
hints.push('the startup log should say "leaving ... GiB of weights on the checkpoint mapping" -- if it does not, the host is not the constraint you set');
hints.push("every figure here is anchored at 480P: activations grow with the pixel count, so at 768P drop the resident DiT layers to 0 first, then video_vae to 24 if the decode still collides -- the flags trade speed for headroom in that order");
hints.push("on a physical 32 GB host the page cache cannot hold the per-step weight sweep, so every step re-reads ~40-65 GB from disk and the drive is the denoise clock: a real desktop 4090 with a 990 Pro measured 38 s/step (52.9 GB read per step). Resident DiT layers cut that read directly (~1 GB/step each), so raise them as far as VRAM allows; 64 GB of RAM caches the sweep and returns to the quoted times");
hints.push("what the drive has to deliver is computable: each step re-reads (50 - resident DiT layers - the layers the host pin budget covered) x 1.23 GiB, and the courier overlaps that read with compute, so the drive stops being the denoise clock once its sequential rate exceeds those bytes divided by this card's step time. A 32 GB host covers ~13 layers, which puts a 24 GB card at ~38 GiB/step and a 12/16 GB card at ~46 GiB/step: a PCIe 4.0 NVMe (6-7 GiB/s) clears both, a PCIe 3.0 drive (~3.2 GiB/s) becomes the clock, and a SATA SSD is not usable for this model");
hints.push("the 38 s/step a physical 4090 with a 990 Pro once measured (52.9 GB per step at ~1.4 GB/s) was the old path: the pinned-store pool rounded every block up to a power of two, so the pin budget over-committed and the machine had to run with pinning off, and the layers that stayed mapped faulted in 4 KiB at a time. Exact-size pins and O_DIRECT reads through the courier removed both -- on the physical 5090 the drive is now fully hidden behind compute. A 4090 on a 32 GB host is derived to return to its compute wall (~8.5 s/step) on a PCIe 4.0 drive; that machine has not been re-measured");
hints.push("resident DiT layers still remove 1.23 GiB/step of drive traffic each, but only while the streamed set is larger than the pin budget; once the read fits under the step's compute time, more resident layers buy nothing and only cost VRAM");
hints.push("warm at the shape you will serve (--warmup-resolutions WxH --warmup-num-frames N): the default warmup is 1344x768 x 124 frames, which cost 7.6 min of startup on the physical 5090 and sizes the residency plan for a request you are not going to make");
hints.push("the pin budget takes 95% of the memory available after loading, so on a machine you also use as a desktop expect swap pressure: the 5090 run above finished with its 8 GB swap file full. There is no supported knob for a smaller budget yet; giving the card more resident layers is the lever that exists, since a resident layer is one the host no longer has to pin");
hints.push("on Windows run under WSL2, and keep the checkpoint inside the ext4 side (under ~), never on /mnt/c -- the NTFS bridge reads an order of magnitude slower and multiplies the disk clock");
return hints;
}
@@ -294,7 +329,7 @@ return {
id: "auto",
label: "Auto",
flags: (s) => {
if (CONSUMER_SINGLE.includes(s.hw)) return consumerFlags(s);
if (consumerSingleCard(s)) return consumerFlags(s);
const recipe = config.commandBuilder.resource.verifiedRecipes.find((entry) =>
entry.hw === s.hw && entry.nodes === Number(s.nodes)
&& entry.gpus_per_node === Number(s.gpus_per_node));
@@ -306,7 +341,7 @@ return {
"--dit-layerwise-resident-layers 20",
] : ["--performance-mode speed"];
},
hints: (s) => (CONSUMER_SINGLE.includes(s.hw) ? consumerHints(s) : []),
hints: (s) => (consumerSingleCard(s) ? consumerHints(s) : []),
description: "Use the recommended placement for the selected hardware and resource shape.",
},
{
@@ -330,14 +365,14 @@ return {
id: "offload",
label: "Layerwise offload",
flags: (s) => {
if (CONSUMER_SINGLE.includes(s.hw)) return consumerFlags(s);
if (consumerSingleCard(s)) return consumerFlags(s);
return [
"--performance-mode memory",
"--layerwise-offload-components dit,text_encoder,vae",
"--dit-layerwise-resident-layers 20",
];
},
hints: (s) => (CONSUMER_SINGLE.includes(s.hw) ? consumerHints(s) : []),
hints: (s) => (consumerSingleCard(s) ? consumerHints(s) : []),
soft: (s) => s.hw !== "rtx5090" && !CONSUMER_SINGLE.includes(s.hw),
softReason: "Tuned and verified on the consumer cards. It runs on the datacenter GPUs too, where a resident recipe is simply faster.",
recommendedWhen: (s) => s.hw === "rtx5090" || CONSUMER_SINGLE.includes(s.hw),