diff --git a/docs/cookbook/diffusion/Wan/Wan2.1.mdx b/docs/cookbook/diffusion/Wan/Wan2.1.mdx index 184de039f7..db8abc4b1d 100644 --- a/docs/cookbook/diffusion/Wan/Wan2.1.mdx +++ b/docs/cookbook/diffusion/Wan/Wan2.1.mdx @@ -45,6 +45,30 @@ Current supported optimization options are listed in the [SGLang diffusion suppo - `--ring-degree {RING_DEGREE}`: Degree of ring attention-style SP in USP. - `--text-encoder-cpu-offload`, `--dit-cpu-offload`, `--vae-cpu-offload`: Use CPU offload to reduce peak GPU memory when needed. +### 3.3 24 GB NVIDIA GPUs + +The 14B DiTs do not fit on a 24 GB card in BF16 (the I2V-14B one is 30.5 GiB), so their weights have to stream from host memory layer by layer. On CUDA, `--performance-mode memory` streams the DiT, text encoder, image encoder and VAE: + +```bash +# 1 GPU +sglang serve --model-path Wan-AI/Wan2.1-I2V-14B-480P-Diffusers --performance-mode memory + +# 8 GPUs: CFG parallel x Ulysses 4 +sglang serve --model-path Wan-AI/Wan2.1-I2V-14B-480P-Diffusers --num-gpus 8 \ + --enable-cfg-parallel --ulysses-degree 4 --performance-mode memory +``` + +Each GPU's worker keeps its own copy of the streamed weights in host memory, about 60 GiB, so 8 GPUs need close to 500 GiB of RAM. + +Measured with SGLang main `c7be3e9` (2026-09-29) on RTX 4090 24 GB (PCIe, no P2P between GPUs), for Wan2.1-I2V-14B-480P image-to-video at 480x832, 81 frames, 40 steps, BF16: + +| GPUs | Time per video | Time per step | Peak GPU memory | Host memory | +|---|---:|---:|---:|---:| +| 1 | 840 s | 20.7 s | 12.5 GiB | 59 GiB | +| 8 | 198 s | 4.8 s | 11.9 GiB | 483 GiB | + +Component CPU offload (`--dit-cpu-offload`) does not help here: it moves the whole DiT onto the GPU around each use. + ## 4. Model Invocation ### 4.1 Basic Usage diff --git a/python/sglang/multimodal_gen/runtime/managers/gpu_worker.py b/python/sglang/multimodal_gen/runtime/managers/gpu_worker.py index 598108554d..778a6e0574 100644 --- a/python/sglang/multimodal_gen/runtime/managers/gpu_worker.py +++ b/python/sglang/multimodal_gen/runtime/managers/gpu_worker.py @@ -1672,20 +1672,24 @@ class GPUWorker(GPUWorkerPostTrainingMixin): OOM_MSG = """ OOM detected. Possible solutions: - - If the OOM occurs during loading: + - If the OOM occurs during loading, or on the first request with a component on + CPU offload (which moves the whole component onto the GPU when it runs): 1. Check available memory on every selected GPU, not only total capacity. In multi-GPU runs, the least-free selected GPU is the bottleneck. - 2. For single-GPU deployment, use `--performance-mode memory`, component CPU offload, - or `--dit-layerwise-offload` for supported Wan/MOVA DiTs. - 3. For multi-GPU deployment, keep the default `--performance-mode auto` or set - `--use-fsdp-inference true` to shard DiT weights with FSDP. FSDP is not a - single-GPU substitute for CPU offload. + 2. For single-GPU deployment, stream weights layer by layer by listing components + in `--layerwise-offload-components`, e.g. `dit,text_encoder,image_encoder,vae`. + `--dit-layerwise-offload` streams only the DiT, and `--performance-mode memory` + streams the DiT only for models that validate it. Component CPU offload helps + only when each component fits on its own. + 3. For multi-GPU deployment, set `--use-fsdp-inference true` to shard DiT weights + with FSDP. FSDP is not a single-GPU substitute for offload. - If the OOM occurs during runtime: 1. Reduce resolution, `--num-frames`, or batch size. 2. Use `--performance-mode memory` for lower memory usage. 3. Enable SP/Ulysses/Ring for sequence-heavy workloads in multi-GPU setups. 4. Use FSDP, with CFG parallelism when supported, for validated multi-GPU workloads. 5. Use a lower-memory attention backend or quantization when available. + Tested launch commands per model: https://docs.sglang.io/cookbook Or, open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose """