diff --git a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx index c9a1755bf..83ad5b723 100644 --- a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx +++ b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx @@ -382,7 +382,27 @@ Poll and download any conditioned request with the same job-status and content endpoints used in the T2VA example. Server-local `file://` URIs must refer to files visible inside the SGLang server environment. -## 5. Sampling and output controls +## 5. Turbo LoRA for few-step generation + +[`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora) distills the native **FL2VA** DiT for usable **4–8 step** generation. On `--model-variant fl2va`, use it with **`t2va`** or **`fl2va`** from section 4 and set `"num_inference_steps": 4` or `8` instead of `50`. **`ref2va`** uses a separate checkpoint partition and is not validated with this LoRA. + +```bash Command +curl -sS -X POST http://127.0.0.1:30010/v1/set_lora \ + -H "Content-Type: application/json" \ + -d '{ + "lora_nickname": "h3-turbo", + "lora_path": "larryvrh/MiniMax-H3-Turbo-Lora", + "strength": 1.0 + }' +``` + +When the repository contains multiple safetensors files, prefer `minimax_h3_turbo_4step_ckpt500.safetensors` (upstream default) via `--lora-path` and `--lora-weight-name` on `sglang serve`, or pass the local path to that file as `lora_path`. + + +LoRAs for the ComfyUI pruned MiniMax-H3 graph (for example H3-GalaxyAce) are not compatible with SGLang's native FL2VA weights. + + +## 6. Sampling and output controls MiniMax-H3 supports more than one output per prompt. The video API accepts `num_outputs_per_prompt` (or OpenAI-compatible `n`) from 1 through 10. Offline @@ -532,7 +552,7 @@ task profile. A real B200 request has completed, but the `quality: "high"` path above remains fail-closed to the audited 4×H200 workload. -## 6. Runtime feature recipes +## 7. Runtime feature recipes @@ -623,7 +643,7 @@ real H3 validation runs. -## 7. Configuration notes +## 8. Configuration notes - MiniMax-H3 produces the canonical 24 fps output; request duration is expressed through `target.duration_seconds`. - `target.duration_seconds` must be between 4 and 15 seconds, inclusive. The command picker defaults to the verified 5-second profile. @@ -642,7 +662,7 @@ real H3 validation runs. - `speed` keeps model components resident, `auto` applies the model-aware 120 GiB residency threshold, and `memory` enables the memory-saving placement policy. Explicit `--layerwise-offload-components` overrides that placement list. DiT residency/prefetch knobs are scoped to the DiT; the text encoder and video VAE decoder use one-layer prefetch and zero residency, while the H3 video VAE encoder stays resident. When `memory` is combined with explicit FSDP, H3 instead keeps the sharded DiT on GPU and layerwise-offloads the text encoder and executable VAE decoder blocks. Use `speed` only after confirming that the complete target workload fits. - Breakable CUDA graph execution is an explicit opt-in, not part of the recommended `speed` preset. It requires `--enable-breakable-cuda-graph`, every served size in `--warmup-resolutions`, and `--bcg-text-buckets` that cover the live H3 condition sequence. The validated 1344×768 Ref2VA recipe uses 5504; other task profiles and reference sets may need a different value. It preserves eager output for matching captured signatures, but graph capture consumes additional GPU memory and may provide little latency benefit when Ulysses attention and collectives dominate, so benchmark it on the target topology before enabling it. -## 8. Benchmarks +## 9. Benchmarks The picker exposes resident and FSDP profiles on NVIDIA datacenter GPUs. GPU counts are properties of the selected recipes, not a claim that every platform diff --git a/docs/docs/sglang-diffusion/compatibility_matrix.mdx b/docs/docs/sglang-diffusion/compatibility_matrix.mdx index 8b90d91b6..9cf49963a 100644 --- a/docs/docs/sglang-diffusion/compatibility_matrix.mdx +++ b/docs/docs/sglang-diffusion/compatibility_matrix.mdx @@ -797,6 +797,10 @@ The entries below simply reflect configurations that have been manually validate + + MiniMax-H3 + `larryvrh/MiniMax-H3-Turbo-Lora` + Wan2.2 `lightx2v/Wan2.2-Distill-Loras`
`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1` diff --git a/docs/docs/sglang-diffusion/quantization.mdx b/docs/docs/sglang-diffusion/quantization.mdx index ea8171c6b..2ff239f05 100644 --- a/docs/docs/sglang-diffusion/quantization.mdx +++ b/docs/docs/sglang-diffusion/quantization.mdx @@ -136,7 +136,7 @@ sglang generate \ MiniMax-H3 supports this path while preserving its required FP32 patch, timestep, and output projections. See the -[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#6-runtime-feature-recipes) +[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-runtime-feature-recipes) for its distributed serving recipe. ### MXFP4 Online Quantization diff --git a/python/sglang/multimodal_gen/runtime/layers/lora/linear.py b/python/sglang/multimodal_gen/runtime/layers/lora/linear.py index ba3ca1571..47023e8f6 100644 --- a/python/sglang/multimodal_gen/runtime/layers/lora/linear.py +++ b/python/sglang/multimodal_gen/runtime/layers/lora/linear.py @@ -523,11 +523,26 @@ class MergedColumnParallelLinearWithLoRA(ColumnParallelLinearWithLoRA): def slice_lora_b_weights(self, B: torch.Tensor) -> torch.Tensor: tp_rank = get_tp_rank() - # Since the outputs for both gate and up are identical, we use a random one. - shard_size = self.base_layer.output_partition_sizes[0] - start_idx = tp_rank * shard_size - end_idx = (tp_rank + 1) * shard_size - return B[:, start_idx:end_idx, :] + if B.dim() == 3: + # Stacked Q/K/V (or gate/up) LoRA weights from diffusers-style adapters. + shard_size = self.base_layer.output_partition_sizes[0] + start_idx = tp_rank * shard_size + end_idx = (tp_rank + 1) * shard_size + return B[:, start_idx:end_idx, :] + + # Native fused checkpoints (MiniMax H3, etc.) store one concatenated 2D + # lora_B matrix per logical layer; shard each section independently. + shards: list[torch.Tensor] = [] + row_offset = 0 + for full_size, part_size in zip( + self.base_layer.output_sizes, + self.base_layer.output_partition_sizes, + ): + local_start = tp_rank * part_size + local_end = (tp_rank + 1) * part_size + shards.append(B[row_offset + local_start : row_offset + local_end, :]) + row_offset += full_size + return torch.cat(shards, dim=0) class QKVParallelLinearWithLoRA(ColumnParallelLinearWithLoRA): diff --git a/python/sglang/multimodal_gen/test/unit/test_lora_format_adapter.py b/python/sglang/multimodal_gen/test/unit/test_lora_format_adapter.py index 219207aa2..a60b7ee01 100644 --- a/python/sglang/multimodal_gen/test/unit/test_lora_format_adapter.py +++ b/python/sglang/multimodal_gen/test/unit/test_lora_format_adapter.py @@ -280,6 +280,18 @@ def _run_all_tests() -> List[Dict]: ) ) + # MiniMax-H3 Turbo LoRA (native diffusers/PEFT-style keys). + results.append( + run_single_test( + name="MiniMax H3 Turbo LoRA", + repo_id="larryvrh/MiniMax-H3-Turbo-Lora", + filename="minimax_h3_turbo_4step.safetensors", + local_name="minimax_h3_turbo_4step.safetensors", + expected_before=LoRAFormat.STANDARD, + expected_after=LoRAFormat.STANDARD, + ) + ) + return results