diff --git a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx index 69fd2aed7..8097a3313 100644 --- a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx +++ b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx @@ -758,7 +758,7 @@ real H3 validation runs. - For throughput-oriented serving, select **DP (batched throughput)**. The picker pairs `--encoder-parallel dp` with an editable `--batching-max-size` greater than 1; compatible requests are distributed across ranks, while every rank keeps a full encoder replica. Encoder DP requires TP1 and DiT DP1, so it is disabled for the H100 TP2 + Ulysses2 and RTX 5090 TP2 recipes. It provides no benefit for a batch of one and is not bitwise-identical to the folded deployment. - Use explicit **Fold** to prioritize single-request latency and encoder memory on a measured high-bandwidth single-node topology. Use **Replicate** as the compatibility path when folding or encoder DP is unsuitable. - `--use-fsdp-inference true` shards only the DiT. MiniMax-H3 preserves the original FP32 dtype of its patch, time, and output projections during FSDP all-gather, so this path does not trade numerical correctness for memory. On 4×H100, prefer TP2 + Ulysses2 for speed; use FSDP as an explicit capacity policy rather than assuming it is faster. -- `speed` keeps model components resident, `auto` applies the model-aware 120 GiB residency threshold, and `memory` enables the memory-saving placement policy. Explicit `--layerwise-offload-components` overrides that placement list. DiT residency/prefetch knobs are scoped to the DiT; the text encoder and video VAE decoder use one-layer prefetch and zero residency, while the H3 video VAE encoder stays resident. When `memory` is combined with explicit FSDP, H3 instead keeps the sharded DiT on GPU and layerwise-offloads the text encoder and executable VAE decoder blocks. Use `speed` only after confirming that the complete target workload fits. +- `speed` keeps model components resident, while `auto` applies the model-aware 120 GiB residency threshold. `memory` prioritizes avoiding OOM and includes the executable VAE decoder in its default layerwise set. A measured recipe with sufficient headroom can opt into `--component-residency vae=resident`; the 2×H100 CI recipe does this because the VAE's 4.8 GiB/GPU cost avoids repeated decoder transfers during tiled decode. DiT residency and prefetch knobs remain scoped to the DiT. Use `speed` only after confirming that the complete target workload fits. - Breakable CUDA graph execution is an explicit opt-in, not part of the recommended `speed` preset. It requires `--enable-breakable-cuda-graph`, every served size in `--warmup-resolutions`, and `--bcg-text-buckets` that cover the live H3 condition sequence. The validated 1344×768 Ref2VA recipe uses 5504; other task profiles and reference sets may need a different value. It preserves eager output for matching captured signatures, but graph capture consumes additional GPU memory and may provide little latency benefit when Ulysses attention and collectives dominate, so benchmark it on the target topology before enabling it. ## 9. Benchmarks diff --git a/docs/docs/sglang-diffusion/api/cli.mdx b/docs/docs/sglang-diffusion/api/cli.mdx index f116e0f52..103a67496 100644 --- a/docs/docs/sglang-diffusion/api/cli.mdx +++ b/docs/docs/sglang-diffusion/api/cli.mdx @@ -235,6 +235,8 @@ sglang generate \ --prompt "A quiet city street after rain" ``` +Values passed to the compatibility option `--layerwise-offload-components` must match loaded component keys, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Its `default` group selects text encoders, image encoders, and VAEs. Use `all` to select every layerwise-offloadable component. + Layerwise tuning options such as `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, and `--dit-layerwise-residency-policy` continue to control the streamed layer working set. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency. ## Serve diff --git a/docs/docs/sglang-diffusion/deployment_cookbook.mdx b/docs/docs/sglang-diffusion/deployment_cookbook.mdx index 70221aba3..624ea1c08 100644 --- a/docs/docs/sglang-diffusion/deployment_cookbook.mdx +++ b/docs/docs/sglang-diffusion/deployment_cookbook.mdx @@ -128,7 +128,7 @@ See [OpenAI API: Served model name](/docs/sglang-diffusion/api/openai_api#served -`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. For image workloads with at least 45 GiB available per selected GPU, it keeps the repeatedly reused DiT resident and uses layerwise offload for large auxiliary encoders; below that threshold it keeps the DiT offloaded. Video DiT residency remains model- and workload-specific because frame count and resolution change its peak memory substantially. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk. It keeps `torch.compile` disabled by default because its effect varies by model and workload. A model-specific deployment config may enable a validated compile path, and `--enable-torch-compile true` always opts in explicitly. +`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. For image workloads with at least 45 GiB available per selected GPU, it keeps the repeatedly reused DiT resident and uses layerwise offload for large auxiliary encoders; below that threshold it keeps the DiT offloaded. Model-specific components such as VAEs become resident only when their configured memory threshold is met. `memory` instead keeps the VAE in its default layerwise set to maximize memory headroom; use `--component-residency vae=resident` for a measured recipe with sufficient capacity. Video DiT residency remains model- and workload-specific because frame count and resolution change its peak memory substantially. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk. It keeps `torch.compile` disabled by default because its effect varies by model and workload. A model-specific deployment config may enable a validated compile path, and `--enable-torch-compile true` always opts in explicitly. The modes tune native pipeline components declared to the component residency manager. DiTs, text/image encoders, VAEs, vocoders, adapters, and upsamplers can use layerwise offload when their native module declares its executable layer structure. Explicitly selecting an unsupported component fails at startup instead of falling back to another residency mode. diff --git a/python/sglang/multimodal_gen/test/server/gpu_cases.py b/python/sglang/multimodal_gen/test/server/gpu_cases.py index 1bd33881c..a02a35727 100644 --- a/python/sglang/multimodal_gen/test/server/gpu_cases.py +++ b/python/sglang/multimodal_gen/test/server/gpu_cases.py @@ -700,7 +700,9 @@ TWO_GPU_CASES = [ "--performance-mode", "memory", "--layerwise-offload-components", - "dit,text_encoder,vae", + "dit,text_encoder", + "--component-residency", + "vae=resident", "--dit-offload-prefetch-size", "1", "--dit-layerwise-resident-layers", diff --git a/python/sglang/multimodal_gen/test/server/perf_baselines/h100.json b/python/sglang/multimodal_gen/test/server/perf_baselines/h100.json index 0d5b6e490..46b3fffcf 100644 --- a/python/sglang/multimodal_gen/test/server/perf_baselines/h100.json +++ b/python/sglang/multimodal_gen/test/server/perf_baselines/h100.json @@ -2808,7 +2808,7 @@ "MiniMaxH3LatentPreparationStage": 18.72, "MiniMaxH3TimestepPreparationStage": 0.34, "MiniMaxH3DenoisingStage": 16839.0, - "MiniMaxH3DecodingStage": 29741.66, + "MiniMaxH3DecodingStage": 2200.0, "per_frame_generation": null }, "denoise_step_ms": { @@ -2820,10 +2820,10 @@ "5": 2343.56, "6": 2326.79 }, - "expected_e2e_ms": 47538.57, + "expected_e2e_ms": 19700.0, "expected_avg_denoise_ms": 2045.61, "expected_median_denoise_ms": 2332.99, - "estimated_full_test_time_s": 235.9 + "estimated_full_test_time_s": 208.0 }, "mova_360p_tp2": { "stages_ms": {}, diff --git a/python/sglang/multimodal_gen/test/unit/test_server_args.py b/python/sglang/multimodal_gen/test/unit/test_server_args.py index dc57db0ae..659cb0734 100644 --- a/python/sglang/multimodal_gen/test/unit/test_server_args.py +++ b/python/sglang/multimodal_gen/test/unit/test_server_args.py @@ -1154,7 +1154,7 @@ class TestOffloadDefaults(unittest.TestCase): with self.assertRaisesRegex(ValueError, "native SGLang backend"): resolve_diffusers_pipeline_offload({"all": LAYERWISE_OFFLOAD}) - def test_vae_cpu_offload_defaults_false_on_low_memory_gpu(self): + def test_memory_mode_layerwise_offloads_vae_on_low_memory_gpu(self): args = self._from_dict_with_task_type( ModelTaskType.T2V, memory_gb=16, @@ -1169,6 +1169,22 @@ class TestOffloadDefaults(unittest.TestCase): args.layerwise_offload_components, ["text_encoder", "image_encoder", "vae"], ) + self.assertEqual(args.residency_mode("vae"), LAYERWISE_OFFLOAD) + + def test_memory_mode_preserves_explicit_vae_residency(self): + for kwargs, expected_mode in ( + ({"component_residency": ["vae=resident"]}, RESIDENT), + ({"vae_cpu_offload": True}, COMPONENT_OFFLOAD), + ): + with self.subTest(expected_mode=expected_mode): + args = self._from_dict_with_task_type( + ModelTaskType.T2V, + memory_gb=16, + kwargs={"performance_mode": "memory", **kwargs}, + ) + + self.assertNotIn("vae", args.layerwise_offload_components or []) + self.assertEqual(args.residency_mode("vae"), expected_mode) def test_explicit_vae_cpu_offload_true_is_preserved_by_default_layerwise( self, @@ -2204,6 +2220,8 @@ class TestOffloadDefaults(unittest.TestCase): self.assertFalse(args.dit_layerwise_offload) self.assertIn("text_encoder", args.layerwise_offload_components or []) self.assertIn("vae", args.layerwise_offload_components or []) + self.assertFalse(args.vae_cpu_offload) + self.assertEqual(args.residency_mode("vae"), LAYERWISE_OFFLOAD) def test_minimax_h3_rejects_explicit_cfg_parallel(self): with self.assertRaisesRegex(