minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[diffusion] Keep Cosmos3 Nano resident on 96 GB GPUs (#36641)

Browse Source
This commit is contained in:
Xiaoyu Zhang
2026-08-29 07:40:46 +08:00
committed by GitHub
parent e1b3bba3cc
commit 50bc1a3767
4 changed files with 103 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/diffusion/Cosmos/Cosmos3.mdx
+8 -7
View File
@@ -62,13 +62,14 @@ sglang serve \
--num-gpus 1
```
With `--performance-mode auto`, a Cosmos3 checkpoint keeps its DiT and VAE
resident when every selected GPU has at least 120 GiB available at startup.
Below that threshold, auto mode retains the conservative DiT
component-offload policy. Cosmos3 runs one DiT per pipeline, so component
offload above the threshold only pays to copy the weights out to host memory
and back on every request. Serve `Cosmos3-Super` across multiple GPUs as
shown below so each rank holds a shard of the weights.
With `--performance-mode auto`, Cosmos3 Nano keeps its DiT and VAE resident
when every selected GPU has at least 90 GiB available at startup. Other
Cosmos3 checkpoints use a 120 GiB threshold. Below the applicable threshold,
auto mode retains the conservative DiT component-offload policy. Cosmos3 runs
one DiT per pipeline, so component offload above the threshold only pays to
copy the weights out to host memory and back on every request. Serve
`Cosmos3-Super` across multiple GPUs as shown below so each rank holds a shard
of the weights.
For `Cosmos3-Super`, split the model across multiple GPUs: