diff --git a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx index 467850be34..a30e4b3ee4 100644 --- a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx +++ b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx @@ -61,6 +61,8 @@ or `MiniMax/MiniMax-H3` on ModelScope. Select the checkpoint variant with `--model-variant`: `fl2va` serves both `t2va` and `fl2va`, while `ref2va` serves reference-conditioned requests. SGLang owns the checkpoint-directory mapping; do not point `--model-path` at a manually downloaded subdirectory. +The same FL2VA / Ref2VA split applies in +[ComfyUI integrated mode](#comfyui-integrated-mode). Review the license and usage terms in the MiniMax-H3 model card before production or commercial use. SGLang support does not grant additional model usage rights. @@ -296,6 +298,27 @@ recommended lossless preset enables it implicitly. An explicit `--enable-torch-compile true` remains available for controlled experiments, but do not use it to generate consistency ground truth. +### ComfyUI integrated mode + +The [ComfyUI plugin](/docs/sglang-diffusion/comfyui) can run H3 as a per-step +DiT backend. ComfyUI keeps CLIP, VAE, and the sampler loop. SGLang loads only +the transformer under `--comfyui-mode`. + +Copy `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion` into ComfyUI +`custom_nodes/`. Load a shipped graph and set `SGLDOptions.model_type` to +`minimax_h3`: + +| Graph | Task | DiT filename | +| --- | --- | --- | +| `minimax_h3_t2v_sgld.json` | T2V / I2VA / FL2VA | `minimax_h3_fl2va_*.safetensors` | +| `minimax_h3_r2v_sgld.json` | Reference-to-video | `minimax_h3_ref2va_*.safetensors` | +| `minimax_h3_t2v_sgld_upscaler.json` | Low-res then 3D ×2 | `fl2va` DiT | + +Pruned BF16 / FP8 files from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models) +are the usual loader inputs. GGUF goes in `SGLDOptions.transformer_weights_path`. +On a 24 GB card put the 32B CLIP on CPU and enable DiT layerwise offload; use +`kitchen_int8` or GGUF rather than the full unpruned BF16 DiT. + ### Singularity hybrid weights [Singularity](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity) is an diff --git a/docs/docs.json b/docs/docs.json index 37c3163a7d..f993070abd 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1654,6 +1654,7 @@ "pages": [ "docs/sglang-diffusion/api/cli", "docs/sglang-diffusion/api/openai_api", + "docs/sglang-diffusion/comfyui", "docs/sglang-diffusion/realtime_models", "docs/sglang-diffusion/models_with_ar", "docs/sglang-diffusion/models_with_pe", diff --git a/docs/docs/sglang-diffusion/comfyui.mdx b/docs/docs/sglang-diffusion/comfyui.mdx new file mode 100644 index 0000000000..f9a13e7005 --- /dev/null +++ b/docs/docs/sglang-diffusion/comfyui.mdx @@ -0,0 +1,51 @@ +--- +title: ComfyUI plugin +description: Use SGLang Diffusion from ComfyUI in server mode or as a per-step DiT backend. +--- + +The [ComfyUI SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion) has two modes. + +**Server mode** talks to a standalone `sglang serve` process over HTTP. ComfyUI sends prompts and receives images or video. This is the same path as the [OpenAI-compatible API](/docs/sglang-diffusion/api/openai_api). + +**Integrated mode** keeps ComfyUI's CLIP, VAE, and sampler loop. SGLang loads only the DiT and runs one forward per sampler step. The worker starts the native Flux / Qwen-Image / Z-Image / MiniMax-H3 pipeline under `--comfyui-mode`. A single-file ComfyUI `.safetensors` (or a GGUF overlay) is loaded through a checkpoint spec. Each `apply_model` call is translated by a per-model adapter. + +Integrated mode does not ship a separate `comfyui_*` pipeline class per model. + +## Install the plugin + +1. Install `sglang[diffusion]`. See [Installation](/docs/sglang-diffusion/installation). +2. Copy `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion` into ComfyUI's `custom_nodes/` directory. +3. Restart ComfyUI. + +Example workflows live next to the plugin under `workflows/`. + +## Integrated mode + +1. Load the DiT with `SGLDiffusion UNET Loader`. +2. Set `num_gpus`, `tp_size`, `model_type`, or compile flags with `SGLDiffusion Options`. +3. Connect the loaded model to a standard ComfyUI sampler. + +Supported integrated-mode families: Flux, Qwen-Image, Z-Image, MiniMax-H3. Qwen-Image edit is experimental. + +## How a sampler step reaches the worker + +The plugin README has the [architecture diagram](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion#architecture). The hop is: + +1. A model adapter packs ComfyUI tensors into an SGLang `Req`. +2. Local ZMQ replaces CUDA tensors with IPC handles so latents stay on GPU. +3. Rank 0 materializes the handles. Multi-rank `--comfyui-mode` then detaches CUDA tensors and broadcasts them over NCCL. The general SP / CFG / TP path is still the original `broadcast_pyobj`. +4. The worker pipeline keeps `transformer` plus a pass-through scheduler. After the first step, conditioning stays in a worker session; later steps send latents and the timestep. + +Integrated mode currently supports: + +| Family | ComfyUI `model_type` | Native pipeline | Example checkpoints | +| --- | --- | --- | --- | +| Flux | `flux` | `FluxPipeline` | `FLUX.1-dev` | +| Z-Image | `lumina2` | `ZImagePipeline` | `Z-Image-Turbo` | +| Qwen-Image | `qwen_image` | `QwenImagePipeline` | `Qwen-Image`, `Qwen-Image-2512` | +| Qwen-Image edit | `qwen_image_edit` | `QwenImageEditPlusPipeline` | `Qwen-Image-Edit-2511` (experimental) | +| MiniMax-H3 | `minimax_h3` | `MiniMaxH3Pipeline` | `minimax_h3_fl2va_*.safetensors`, `minimax_h3_ref2va_*.safetensors` | + +MiniMax-H3 example graphs: `minimax_h3_t2v_sgld.json` (T2V / I2VA / FL2VA on the FL2VA DiT), `minimax_h3_r2v_sgld.json` (reference-to-video; use a `ref2va` file), `minimax_h3_t2v_sgld_upscaler.json` (two-pass 3D ×2). Set `SGLDOptions.model_type` to `minimax_h3`. On 24 GB, keep the 32B CLIP on CPU and prefer pruned BF16, GGUF, or `kitchen_int8` with DiT layerwise offload. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#comfyui-integrated-mode). + +To add a model, register a checkpoint spec under `runtime/loader/comfyui_checkpoints/` and a `ComfyUIModelAdapter`. Do not add another ComfyUI pipeline class. diff --git a/docs/docs/sglang-diffusion/index.mdx b/docs/docs/sglang-diffusion/index.mdx index 69af9138f9..b6879791de 100644 --- a/docs/docs/sglang-diffusion/index.mdx +++ b/docs/docs/sglang-diffusion/index.mdx @@ -33,6 +33,7 @@ sglang serve --model-path Qwen/Qwen-Image --port 30010 - [Supported Models](/docs/sglang-diffusion/compatibility_matrix): browse supported model families, tasks, and public checkpoints - [CLI](/docs/sglang-diffusion/api/cli): run one-off generation jobs or launch a persistent server - [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server +- [ComfyUI plugin](/docs/sglang-diffusion/comfyui): use SGLang from ComfyUI in server mode or as a per-step DiT backend - [Performance Overview](/docs/sglang-diffusion/performance-optimization): choose speed, memory, parallelism, caching, and quality-tradeoff levers - [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT, TeaCache, or Spectrum to reduce denoising cost - [Quantization](/docs/sglang-diffusion/quantization): configure component checkpoint and causal KV-cache quantization diff --git a/python/sglang/multimodal_gen/README.md b/python/sglang/multimodal_gen/README.md index e14fbfb197..98ad779c83 100644 --- a/python/sglang/multimodal_gen/README.md +++ b/python/sglang/multimodal_gen/README.md @@ -11,7 +11,7 @@ SGLang diffusion features an end-to-end unified pipeline for accelerating diffus SGLang Diffusion has the following features: - Broad model support: Wan, FastWan, FLUX, Qwen-Image / Qwen-Image 2.1, LongCat-Image, Z-Image, Anima, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3/LTX-2.5, MiniMax-H3, FastH3, VDN-H3, LingBot Video MoE, LingBot World, SANA-Video/SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more - Fast inference speed: empowered by optimized `sgl-kernel` kernels, scheduler/runtime improvements, caching acceleration, and native diffusion hot-path optimizations - - Ease of use: OpenAI-compatible api, CLI, and python sdk support + - Ease of use: OpenAI-compatible api, CLI, python sdk, and a [ComfyUI plugin](apps/ComfyUI_SGLDiffusion/README.md) - Multi-platform support: - NVIDIA GPUs (H100, H200, A100, B200, 4090, 5090) - AMD GPUs (MI300X, MI325X, MI355X) diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/README.md b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/README.md index 1fb059efed..13debc2dda 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/README.md +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/README.md @@ -1,6 +1,14 @@ # ComfyUI SGLDiffusion Plugin -A ComfyUI plugin for integrating with SGLang Diffusion server, supporting image and video generation capabilities. +A ComfyUI plugin for SGLang Diffusion. Server mode talks to a standalone HTTP +server. Integrated mode keeps ComfyUI's CLIP / VAE / sampler loop and uses +SGLang only as a per-step DiT forward. + +Integrated mode no longer ships dedicated `comfyui_*` pipelines. It starts the +native Flux / Qwen-Image / Z-Image / MiniMax-H3 pipeline under `--comfyui-mode`, +loads a single-file ComfyUI `.safetensors` (or GGUF overlay) through a +checkpoint spec, and translates each `apply_model` call through a small +per-model adapter. ## Installation @@ -16,7 +24,7 @@ The plugin supports two modes of operation: **Server Mode** (via HTTP API) and * - **Z-Image**: High-speed image generation models (e.g., `Z-Image-Turbo`) - **FLUX**: State-of-the-art text-to-image models (e.g., `FLUX.1-dev`) - **Qwen-Image**: Multi-modal image generation models (e.g., `Qwen-Image`,`Qwen-Image-2512`). *Note: Image editing support is currently experimental and may have some issues.* -- **MiniMax-H3**: Joint video-and-audio generation, server mode only (`SGLDiffusion Generate H3`) +- **MiniMax-H3**: Joint video-and-audio DiT (`model_type=minimax_h3`). Integrated mode: T2V / I2VA / FL2VA use an `fl2va` checkpoint; R2V needs `ref2va`. CLIP and VAE stay in ComfyUI. Server mode uses `SGLDiffusion Generate MiniMax-H3`. ### Mode 1: Server Mode (HTTP API) Connect to a standalone SGLang Diffusion server. @@ -26,14 +34,17 @@ Connect to a standalone SGLang Diffusion server. 3. **Generate Content**: - `SGLDiffusion Generate Image`: For text-to-image and image editing. - `SGLDiffusion Generate Video`: For text-to-video and image-to-video. + - `SGLDiffusion Generate MiniMax-H3`: For joint video-and-audio on a standalone server. 4. **LoRA Support**: Use `SGLDiffusion Server Set LoRA` and `SGLDiffusion Server Unset LoRA`. ### Mode 2: Integrated Mode (Tight Integration) -Leverage SGLang's high-performance sampling directly within ComfyUI while using ComfyUI's front-end nodes (CLIP, VAE, etc.). +ComfyUI keeps CLIP, VAE, and the sampler loop. SGLang loads only the DiT and +runs one forward per sampler step (pass-through scheduler, no text encode / +decode on the worker). 1. **Load Model**: Use the `SGLDiffusion UNET Loader` node to load your diffusion model. 2. **Configure Options**: Use the `SGLDiffusion Options` node to set runtime parameters like `num_gpus`, `tp_size`, `model_type`, or `enable_torch_compile`. -3. **Sample**: Connect the loaded model to standard ComfyUI samplers. SGLang will handle the sampling process efficiently. +3. **Sample**: Connect the loaded model to standard ComfyUI samplers. Each step is packed by a model adapter and sent to the SGLang scheduler. 4. **LoRA Support**: Use the `SGLDiffusion LoRA Loader` for native LoRA integration. ## Adding a Model @@ -74,6 +85,9 @@ Reference workflow files are provided in the `workflows/` directory: - **`z-image_sgld.json`**: High-speed image generation using Z-Image. - **`sgld_text2img.json`**: Server-mode text-to-image generation with LoRA support. - **`sgld_image2video.json`**: Server-mode image-to-video generation. +- **`minimax_h3_t2v_sgld.json`**: MiniMax-H3 T2V / I2VA / FL2VA (`fl2va` DiT). +- **`minimax_h3_r2v_sgld.json`**: MiniMax-H3 reference-to-video (`ref2va` DiT). +- **`minimax_h3_t2v_sgld_upscaler.json`**: H3 two-pass latent upscale (low-res then 3D ×2 refine). For other workflows supporting the models, you can easily use SGLang by replacing the official `UNET Loader` node with the `SGLDUNETLoader` node. Similarly, for LoRA support, replace the official LoRA loader with the `SGLDiffusion LoRA Loader`. @@ -85,4 +99,45 @@ To use these workflows: ## Current Implementation -This plugin provides a high-performance backend for diffusion models in ComfyUI. By leveraging SGLang's optimized kernels and parallelization techniques (Tensor Parallelism, TeaCache, etc.), it significantly accelerates the sampling process, especially for large models like FLUX. +SGLang's optimized kernels and parallelism (TP / SP, compile, cache) run on the +DiT only. Text encoding and VAE stay in ComfyUI. + +## Architecture + +```mermaid +flowchart LR + subgraph comfy [ComfyUI process] + CLIP[CLIP / text encode] + VAE[VAE] + SAMPLER[Sampler loop] + EXEC["SGLDiffusionExecutor"] + ADAPT["Model adapter
pack / unpack"] + CLIP --> SAMPLER + SAMPLER --> EXEC --> ADAPT + end + + ADAPT -->|CUDA IPC spill| R0 + + subgraph sgl [SGLang] + R0[Rank-0 scheduler] + R0 -->|comfyui_mode and multi-rank| NCCL["Detach CUDA tensors
NCCL broadcast"] + R0 -->|otherwise| PYO["Original SP / CFG / TP
broadcast_pyobj"] + NCCL --> PIPE + PYO --> PIPE + PIPE["Native pipeline
--comfyui-mode"] + SPEC["Checkpoint spec
single .safetensors"] --> PIPE + PIPE --> STAGE["Latent prep + session cache
+ DenoisingStage"] + end + + STAGE -->|noise_pred IPC| EXEC + STAGE --> VAE +``` + +Per sampler step: + +1. The adapter turns ComfyUI `apply_model` tensors into an SGLang `Req`. +2. Local ZMQ pickle replaces CUDA tensors with IPC handles so latents stay on GPU. +3. Rank 0 materializes the handles. Multi-rank `--comfyui-mode` then detaches CUDA tensors and broadcasts them over NCCL; the general SP / CFG / TP path is still the original whole-list `broadcast_pyobj`. +4. The worker pipeline is the native model class with modules trimmed to `transformer` + pass-through scheduler. A single-file checkpoint goes through `comfyui_checkpoints`. After the first step, conditioning stays in a worker session; later steps send latents and the timestep. + +Adding a model means a checkpoint spec plus a `ComfyUIModelAdapter`. There is no extra ComfyUI pipeline class. diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/__init__.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/__init__.py index a7d01106b3..2b740a485a 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/__init__.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/__init__.py @@ -4,7 +4,6 @@ Provides generator, model patcher, and server API client. """ from .generator import SGLDiffusionGenerator -from .model_patcher import SGLDModelPatcher from .server_api import SGLDiffusionServerAPI __all__ = [ @@ -12,3 +11,12 @@ __all__ = [ "SGLDModelPatcher", "SGLDiffusionServerAPI", ] + + +def __getattr__(name): + # ModelPatcher subclasses ComfyUI and is only needed when a graph loads a DiT. + if name == "SGLDModelPatcher": + from .model_patcher import SGLDModelPatcher + + return SGLDModelPatcher + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py index e5f61607d0..27a12af829 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py @@ -2,20 +2,103 @@ Generator for SGLang Diffusion ComfyUI integration. """ +import atexit import logging import os -import psutil -from comfy import model_detection, model_management -from comfy.utils import ( - calculate_parameters, - load_torch_file, - state_dict_prefix_replace, - unet_to_diffusers, -) +from ..executors.flux import FluxExecutor +from ..executors.minimax_h3 import MiniMaxH3Executor +from ..executors.zimage import ZImageExecutor logger = logging.getLogger(__name__) + +class _HeaderTensor: + """Shape-only stand-in so detect_unet_config does not load 66GB weights.""" + + def __init__(self, shape): + import torch + + self.shape = torch.Size(shape) + + def numel(self): + n = 1 + for dim in self.shape: + n *= int(dim) + return n + + nelement = numel + + +def _looks_like_gguf(path: str) -> bool: + if not path: + return False + return path.lower().endswith(".gguf") or ( + ":" in path and path.count("/") == 1 and not os.path.isabs(path) + ) + + +def _h3_detect_companion(gguf_ref: str) -> str: + """ComfyUI detect needs H3 safetensors keys; GGUF only overrides the DiT.""" + name = os.path.basename(gguf_ref).lower() + prefer = [] + if "ref2va" in name: + prefer.append("minimax_h3_ref2va_bf16.safetensors") + prefer.extend( + [ + "minimax_h3_fl2va_bf16.safetensors", + "minimax_h3_ref2va_bf16.safetensors", + ] + ) + folders = ( + [os.path.dirname(os.path.abspath(gguf_ref))] + if os.path.dirname(gguf_ref) + else [] + ) + try: + import folder_paths + + folders.extend(folder_paths.get_folder_paths("diffusion_models")) + except Exception: + pass + seen: set[str] = set() + for folder in folders: + if not folder: + continue + for candidate in prefer: + path = os.path.join(folder, candidate) + if path in seen: + continue + seen.add(path) + if os.path.isfile(path): + return path + raise ValueError( + "GGUF transformer needs a MiniMax-H3 BF16 safetensors in " + "models/diffusion_models for ComfyUI architecture detect " + f"(looked for {prefer}). Keep --model-path / unet_name on the " + "BF16 file and pass the GGUF via transformer_weights_path." + ) + + +def _load_state_dict_for_detection(model_path: str): + if _looks_like_gguf(model_path): + raise ValueError( + f"Cannot detect architecture from GGUF {model_path!r}. " + "Use a MiniMax-H3 BF16 safetensors for detect." + ) + if model_path.endswith(".safetensors"): + from safetensors import safe_open + + sd = {} + with safe_open(model_path, framework="pt", device="cpu") as handle: + for key in handle.keys(): + sd[key] = _HeaderTensor(handle.get_slice(key).get_shape()) + return sd + from comfy.utils import load_torch_file + + return load_torch_file(model_path) + + try: from sglang.multimodal_gen import DiffGenerator except ImportError: @@ -23,39 +106,78 @@ except ImportError: "Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'" ) -from ..executors import ( - FluxExecutor, - QwenImageEditExecutor, - QwenImageExecutor, - ZImageExecutor, -) -from .model_patcher import SGLDModelPatcher + +def _load_executor_classes(): + """Qwen adapters import ComfyUI. Keep them optional so CI can load the rest.""" + classes = [FluxExecutor, ZImageExecutor, MiniMaxH3Executor] + try: + from ..executors.qwen_image import QwenImageEditExecutor, QwenImageExecutor + except ModuleNotFoundError as exc: + missing = exc.name or "" + if not missing.startswith("comfy"): + raise + logger.info("ComfyUI is not installed; Qwen image executors are unavailable") + else: + classes.extend([QwenImageExecutor, QwenImageEditExecutor]) + return tuple(classes) + + +_EXECUTOR_CLASSES = _load_executor_classes() + + +_SHARED = None +_ATEXIT_REGISTERED = False class SGLDiffusionGenerator: - """Generator for SGLang Diffusion models in ComfyUI.""" + """Process-wide owner of the one live SGLD worker. + + ComfyUI keeps one ``SGLDUNETLoader`` per node and may reuse a cached + ``MODEL`` after another graph has already replaced the worker. + ``sync_scheduler_client`` is also a process singleton, so a second + ``DiffGenerator`` cannot coexist. All loaders share this object; a + stale executor rebinds itself on the next forward. + """ def __init__(self): self.model_path = None self.generator = None self.executor = None self.last_options = None + self._patcher = None - self.pipeline_class_dict = { - "flux": "ComfyUIFluxPipeline", - "lumina2": "ComfyUIZImagePipeline", # zimage - "qwen_image": "ComfyUIQwenImagePipeline", - "qwen_image_edit": "ComfyUIQwenImageEditPipeline", - } - self.executor_class_dict = { - "flux": FluxExecutor, - "lumina2": ZImageExecutor, - "qwen_image": QwenImageExecutor, - "qwen_image_edit": QwenImageEditExecutor, - } + # Native pipelines, run under comfyui_mode as a DiT-only forward service. + self.pipeline_class_dict = {} + self.executor_class_dict = {} + for executor_cls in _EXECUTOR_CLASSES: + for model_type in executor_cls.adapter_cls.model_types: + self.executor_class_dict[model_type] = executor_cls + self.pipeline_class_dict[model_type] = ( + executor_cls.adapter_cls.pipeline_class_name + ) - def __del__(self): - self.close_generator() + @classmethod + def shared(cls): + global _SHARED, _ATEXIT_REGISTERED + if _SHARED is None: + _SHARED = cls() + if not _ATEXIT_REGISTERED: + atexit.register(cls._atexit_close) + _ATEXIT_REGISTERED = True + return _SHARED + + @classmethod + def reset_shared(cls): + """Test helper: drop the process singleton after closing it.""" + global _SHARED + if _SHARED is not None: + _SHARED.close_generator() + _SHARED = None + + @classmethod + def _atexit_close(cls): + if _SHARED is not None: + _SHARED.close_generator() def init_generator( self, model_path: str, pipeline_class_name: str, kwargs: dict = None @@ -66,7 +188,13 @@ class SGLDiffusionGenerator: if kwargs is None: kwargs = {} # Set comfyui_mode for ComfyUI integration + kwargs = dict(kwargs) kwargs["comfyui_mode"] = True + # ComfyUI already keeps CLIP/VAE in the parent process. Auto image + # policy otherwise sets dit_cpu_offload=True and every sampler step + # reloads the DiT from CPU. + kwargs.setdefault("dit_cpu_offload", False) + kwargs = self._server_args_kwargs(kwargs) self.generator = DiffGenerator.from_pretrained( model_path=model_path, pipeline_class_name=pipeline_class_name, @@ -74,62 +202,132 @@ class SGLDiffusionGenerator: ) return self.generator - def kill_generator(self): - """Kill worker processes manually because generator shutdown cannot terminate them.""" - current_pid = os.getpid() - worker_processes = [] - for proc in psutil.process_iter(["pid", "name", "cmdline"]): - try: - # Look for sglang-diffusionWorker processes - if proc.info["cmdline"]: - cmdline = " ".join(proc.info["cmdline"]) - if "sgl_diffusion::" in cmdline: - if proc.info["pid"] != current_pid: - worker_processes.append(proc) - except (psutil.NoSuchProcess, psutil.AccessDenied): - continue + @staticmethod + def _server_args_kwargs(kwargs: dict) -> dict: + """Drop plugin-only / stale flags that ServerArgs no longer accepts.""" + import dataclasses - if worker_processes: - logger.info( - f"Found {len(worker_processes)} worker processes to terminate..." + from sglang.multimodal_gen.runtime.server_args import ServerArgs + + valid = {f.name for f in dataclasses.fields(ServerArgs)} + aliases = {"dp_degree": "dp_size", "cache_strategy": None, "model_type": None} + cleaned = {} + for key, value in kwargs.items(): + dest = aliases.get(key, key) + if dest is None or dest not in valid: + continue + cleaned[dest] = value + return cleaned + + def _worker_processes(self): + gen = self.generator + if gen is None: + return [] + return list(getattr(gen, "local_scheduler_process", None) or []) + + def _is_live(self) -> bool: + """True when this owner still has a usable worker + scheduler client.""" + if self.generator is None: + return False + try: + from sglang.multimodal_gen.runtime.scheduler_client import ( + sync_scheduler_client, ) - for proc in worker_processes: - try: - logger.info( - f"Terminating worker process {proc.info['pid']}: {proc.info['name']}" - ) - proc.terminate() - proc.wait(timeout=5) - except psutil.TimeoutExpired: - logger.warning( - f"Process {proc.info['pid']} did not terminate, forcing kill..." - ) - try: - proc.kill() - proc.wait(timeout=2) - except (psutil.NoSuchProcess, psutil.TimeoutExpired): - pass - except (psutil.NoSuchProcess, psutil.AccessDenied): - pass + except ImportError: + return False + ctx = getattr(sync_scheduler_client, "context", None) + if ctx is None or getattr(ctx, "closed", False): + return False + workers = self._worker_processes() + if workers and not all(proc.is_alive() for proc in workers): + return False + return True + + def _can_reuse(self, gather_options: dict) -> bool: + return ( + self.last_options is not None + and self.last_options == gather_options + and self.generator is not None + and getattr(self, "_patcher", None) is not None + and self.executor is not None + and self._is_live() + ) + + def _owns_live(self, executor) -> bool: + return ( + executor is not None + and self.generator is not None + and executor.generator is self.generator + and self._is_live() + ) + + def ensure_executor(self, executor) -> None: + """Rebuild the worker if this cached executor no longer owns it.""" + if self._owns_live(executor): + return + reload = getattr(executor, "_sgld_reload", None) + if not reload: + raise RuntimeError( + "SGLD worker is down and this executor has no reload spec" + ) + logger.info( + "Rebuilding SGLD worker for %s (previous worker was replaced or died)", + reload.get("model_path"), + ) + self.load_model(**reload) + executor.generator = self.generator + lora = getattr(executor, "_lora_input", None) + if lora and lora.get("lora_nickname"): + executor.set_lora(**lora) + + def kill_generator(self): + """Force-stop workers this owner started. Do not scan the process table.""" + for proc in self._worker_processes(): + if proc is None: + continue + try: + if not proc.is_alive(): + continue + logger.info("Terminating SGLD worker pid=%s", getattr(proc, "pid", "?")) + proc.terminate() + proc.join(timeout=5) + if proc.is_alive(): + proc.kill() + proc.join(timeout=2) + except Exception: + logger.warning( + "Failed to terminate SGLD worker %r", proc, exc_info=True + ) def close_generator(self): """Close and cleanup the generator and all associated resources.""" if self.generator is not None: - self.generator.shutdown() + try: + self.generator.shutdown() + except Exception: + logger.warning("DiffGenerator.shutdown failed", exc_info=True) self.kill_generator() - # Clear other references - self.last_options = None - self.model_path = None - self.generator = None - self.executor = None + self.last_options = None + self.model_path = None + self.generator = None + self.executor = None + self._patcher = None def get_comfyui_model(self, model_path: str, model_options: dict = None): """Get ComfyUI model from model path.""" if model_options is None: model_options = {} dtype = model_options.get("dtype", None) - # Allow loading unets from checkpoint files - sd = load_torch_file(model_path) + # Header-only read: H3 BF16 is ~66GB and we only need keys + shapes + # for detect_unet_config. SGLD loads the real weights later. + from comfy import model_detection, model_management + from comfy.utils import ( + calculate_parameters, + state_dict_prefix_replace, + unet_to_diffusers, + ) + + sd = _load_state_dict_for_detection(model_path) diffusion_model_prefix = model_detection.unet_prefix_from_state_dict(sd) temp_sd = state_dict_prefix_replace( sd, {diffusion_model_prefix: ""}, filter_keys=True @@ -185,47 +383,78 @@ class SGLDiffusionGenerator: self, model_path: str, model_options: dict = None, sgld_options: dict = None ): """Load model and return model patcher.""" - gather_options = { + raw_sgld_options = dict(sgld_options) if sgld_options else {} + reload_kwargs = { "model_path": model_path, + "model_options": dict(model_options) if model_options else {}, + "sgld_options": raw_sgld_options, + } + sgld_options = dict(raw_sgld_options) + model_options = dict(reload_kwargs["model_options"]) + plugin_flags = {} + if "enable_cache_dit" in sgld_options: + plugin_flags["enable_cache_dit"] = sgld_options.pop("enable_cache_dit") + set_model_type = sgld_options.pop("model_type", None) + override = (sgld_options.get("transformer_weights_path") or "").strip() + if override: + sgld_options["transformer_weights_path"] = override + elif _looks_like_gguf(model_path): + sgld_options["transformer_weights_path"] = model_path + else: + sgld_options.pop("transformer_weights_path", None) + detect_path = model_path + if _looks_like_gguf(model_path): + detect_path = _h3_detect_companion(sgld_options["transformer_weights_path"]) + gather_options = { + "model_path": detect_path, "model_options": model_options, "sgld_options": sgld_options, + "set_model_type": set_model_type, } - if ( - self.last_options is not None - and self.last_options == gather_options - and self.generator is not None - ): - return self.generator - else: - self.close_generator() + if self._can_reuse(gather_options): + self.executor.enable_cache_dit = plugin_flags.get("enable_cache_dit") + return self._patcher + self.close_generator() self.last_options = gather_options - self.model_path = model_path + self.model_path = detect_path comfyui_model, model_config, model_type = self.get_comfyui_model( - model_path, model_options + detect_path, model_options ) if model_type is None or model_type not in self.pipeline_class_dict: raise ValueError(f"Unsupported model type: {model_type}") - - set_model_type = sgld_options.pop("model_type", None) if sgld_options else None if set_model_type is not None and set_model_type in self.pipeline_class_dict: model_type = set_model_type pipeline_class_name = self.pipeline_class_dict[model_type] self.generator = self.init_generator( - model_path, pipeline_class_name, sgld_options + detect_path, pipeline_class_name, sgld_options ) executor_class = self.executor_class_dict[model_type] self.executor = executor_class( - self.generator, model_path, comfyui_model, model_config + self.generator, detect_path, comfyui_model, model_config ) + self.executor.enable_cache_dit = plugin_flags.get("enable_cache_dit") + self.executor._sgld_reload = reload_kwargs + self.executor._ensure_runtime = self.ensure_executor comfyui_model.diffusion_model = self.executor + from comfy import model_management + from comfy.patcher_extension import WrappersMP + + from .model_patcher import SGLDModelPatcher + load_device = model_management.get_torch_device() offload_device = model_management.unet_offload_device() - return SGLDModelPatcher( + self._patcher = SGLDModelPatcher( comfyui_model, load_device, offload_device, model_type=model_type ) + self._patcher.add_wrapper_with_key( + WrappersMP.SAMPLER_SAMPLE, + "sgld_session", + self.executor.sampler_sample_wrapper, + ) + return self._patcher diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/model_patcher.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/model_patcher.py index c88efd29f0..d146f2b3f0 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/model_patcher.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/model_patcher.py @@ -48,6 +48,17 @@ class SGLDModelPatcher(ModelPatcher): n.backup = self.backup n.object_patches_backup = self.object_patches_backup n.lora_cache = copy.copy(self.lora_cache) + n.model_type = self.model_type + n.wrappers = {} + for wrapper_type, keyed in self.wrappers.items(): + n.wrappers[wrapper_type] = { + key: wrappers.copy() for key, wrappers in keyed.items() + } + n.callbacks = {} + for call_type, keyed in self.callbacks.items(): + n.callbacks[call_type] = { + key: callbacks.copy() for key, callbacks in keyed.items() + } return n def model_size(self): diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/__init__.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/__init__.py index 84afba8425..a6a4a068be 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/__init__.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/__init__.py @@ -3,15 +3,50 @@ ComfyUI SGLang Diffusion executors package. Provides executor classes for different model types. """ +from .adapter import ComfyUIModelAdapter, PackedForward, get_adapter_class from .base import SGLDiffusionExecutor -from .flux import FluxExecutor -from .qwen_image import QwenImageEditExecutor, QwenImageExecutor -from .zimage import ZImageExecutor +from .flux import FluxAdapter, FluxExecutor +from .minimax_h3 import MiniMaxH3Adapter, MiniMaxH3Executor +from .zimage import ZImageAdapter, ZImageExecutor + +# Qwen adapters import ComfyUI (`comfy.ldm.common_dit`). Keep that optional so +# unit tests and SGLD-only paths can load Flux / Z-Image without ComfyUI. __all__ = [ + "ComfyUIModelAdapter", + "PackedForward", "SGLDiffusionExecutor", + "FluxAdapter", "FluxExecutor", + "MiniMaxH3Adapter", + "MiniMaxH3Executor", + "ZImageAdapter", "ZImageExecutor", "QwenImageExecutor", "QwenImageEditExecutor", + "get_adapter_class", ] + + +def __getattr__(name): + if name in { + "QwenImageExecutor", + "QwenImageEditExecutor", + "QwenImageAdapter", + "QwenImageEditAdapter", + }: + from .qwen_image import ( + QwenImageAdapter, + QwenImageEditAdapter, + QwenImageEditExecutor, + QwenImageExecutor, + ) + + mapping = { + "QwenImageAdapter": QwenImageAdapter, + "QwenImageEditAdapter": QwenImageEditAdapter, + "QwenImageExecutor": QwenImageExecutor, + "QwenImageEditExecutor": QwenImageEditExecutor, + } + return mapping[name] + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/adapter.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/adapter.py new file mode 100644 index 0000000000..55fd89a5c2 --- /dev/null +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/adapter.py @@ -0,0 +1,87 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Per-model adapters for the ComfyUI DiT-forward contract. + +The shared executor owns request construction and the ZMQ round-trip. +Each adapter only translates between ComfyUI's ``apply_model`` tensors and +the fields SGLang's ``Req`` expects. Design the interface around H3 (nested +latents, structured payload), not Flux's three-tensor case. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +import torch + +_ADAPTERS: dict[str, type[ComfyUIModelAdapter]] = {} + + +@dataclass +class PackedForward: + """One ComfyUI sampler step, already translated into SGLang tensors.""" + + latents: torch.Tensor + timesteps: torch.Tensor + prompt_embeds: list[torch.Tensor] + height: int + width: int + guidance_scale: float = 1.0 + prompt_seq_lens: list[list[int]] | None = None + pooled_embeds: list[torch.Tensor] | None = None + extra_req: dict[str, Any] = field(default_factory=dict) + unpack_ctx: dict[str, Any] = field(default_factory=dict) + + +class ComfyUIModelAdapter: + """Model-specific pack / unpack / fill_req for one ComfyUI DiT family.""" + + model_types: tuple[str, ...] = () + pipeline_class_name: str = "" + + def __init_subclass__(cls, **kwargs): + super().__init_subclass__(**kwargs) + for model_type in cls.model_types: + _ADAPTERS[model_type] = cls + + def pack( + self, x: torch.Tensor, timestep: torch.Tensor, context, **kwargs + ) -> PackedForward: + raise NotImplementedError + + def unpack( + self, noise_pred: torch.Tensor, packed: PackedForward, x: torch.Tensor + ) -> torch.Tensor: + return noise_pred.to(x.device) + + def fill_req(self, req, packed: PackedForward) -> None: + req.latents = packed.latents + req.timesteps = packed.timesteps + req.prompt_embeds = packed.prompt_embeds + req.raw_latent_shape = torch.tensor(packed.latents.shape, dtype=torch.long) + req.do_classifier_free_guidance = False + if packed.prompt_seq_lens is not None: + req.prompt_seq_lens = packed.prompt_seq_lens + if packed.pooled_embeds is not None: + req.pooled_embeds = packed.pooled_embeds + for key, value in packed.extra_req.items(): + setattr(req, key, value) + + def drop_cached_fields(self, packed: PackedForward) -> None: + packed.prompt_embeds = [] + packed.prompt_seq_lens = None + packed.pooled_embeds = None + packed.extra_req.pop("image_latent", None) + + +def get_adapter_class(model_type: str) -> type[ComfyUIModelAdapter]: + if model_type not in _ADAPTERS: + raise ValueError( + f"Unsupported ComfyUI model type {model_type!r}. " + f"Registered: {sorted(_ADAPTERS)}" + ) + return _ADAPTERS[model_type] + + +def registered_model_types() -> list[str]: + return sorted(_ADAPTERS) diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/base.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/base.py index e14f8226b8..6db631867a 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/base.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/base.py @@ -2,11 +2,23 @@ Base executor class for SGLang Diffusion ComfyUI integration. """ +import uuid + import torch +try: + from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams + from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request +except ImportError: + print( + "Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'" + ) + class SGLDiffusionExecutor(torch.nn.Module): - """Base executor class for SGLang Diffusion models in ComfyUI.""" + """Shared ComfyUI DiT-forward executor. Per-model logic lives on the adapter.""" + + adapter_cls = None def __init__(self, generator, model_path, model, config): super(SGLDiffusionExecutor, self).__init__() @@ -16,6 +28,15 @@ class SGLDiffusionExecutor(torch.nn.Module): self.dtype = config.unet_config["dtype"] self.config = config self.loras = [] + self._lora_input = None + self._sgld_reload = None + self._ensure_runtime = None + if self.adapter_cls is None: + raise TypeError(f"{type(self).__name__} must set adapter_cls") + self.adapter = self.adapter_cls() + self.session_id = uuid.uuid4().hex + self._run_id = 0 + self._sent_conds: set[tuple] = set() @staticmethod def should_suppress_logs(timestep): @@ -26,7 +47,13 @@ class SGLDiffusionExecutor(torch.nn.Module): def set_lora(self, lora_nickname=None, lora_path=None, strength=None, target=None): """Set LoRA adapter using SGLang Diffusion API.""" - if len(lora_nickname) > 0: + self._lora_input = { + "lora_nickname": lora_nickname, + "lora_path": lora_path, + "strength": strength, + "target": target, + } + if lora_nickname and len(lora_nickname) > 0: self.generator.set_lora( lora_nickname=lora_nickname, lora_path=lora_path, @@ -34,23 +61,87 @@ class SGLDiffusionExecutor(torch.nn.Module): target=target, ) - def _unpack_latents(self, latents, height, width, channels): - """Unpack latents from packed format to standard format.""" - batch_size = latents.shape[0] - latents = latents.view(batch_size, height // 2, width // 2, channels, 2, 2) - latents = latents.permute(0, 3, 1, 4, 2, 5) - latents = latents.reshape(batch_size, channels, height, width) + def begin_sampler_run(self) -> None: + """One ComfyUI ``sampler.sample()`` invocation is one cache lifetime.""" + self._run_id += 1 + self._sent_conds = set() - return latents + def end_sampler_run(self) -> None: + """Run cache is evicted on the next bind of a newer id for this executor.""" - def _pack_latents(self, latents): - """Pack latents from standard format to packed format.""" - batch_size, num_channels_latents, height, width = latents.shape - latents = latents.view( - batch_size, num_channels_latents, height // 2, 2, width // 2, 2 + def sampler_sample_wrapper(self, executor, *args, **kwargs): + self.begin_sampler_run() + try: + return executor(*args, **kwargs) + finally: + self.end_sampler_run() + + def comfyui_session_id(self) -> str: + return f"{self.session_id}:{self._run_id}" + + def _cond_key(self, packed) -> tuple | None: + embeds = packed.prompt_embeds + if not embeds: + return None + tensor = embeds[0] + if not torch.is_tensor(tensor) or tensor.numel() == 0: + return None + flat = tensor.reshape(-1) + return ( + tuple(int(dim) for dim in tensor.shape), + float(flat[0].item()), + float(flat[-1].item()), ) - latents = latents.permute(0, 2, 4, 1, 3, 5) - latents = latents.reshape( - batch_size, (height // 2) * (width // 2), num_channels_latents * 4 + + def _mark_and_maybe_drop(self, packed) -> None: + key = self._cond_key(packed) + if key is not None: + packed.extra_req["comfyui_cond_key"] = repr(key) + if key in self._sent_conds: + self.adapter.drop_cached_fields(packed) + else: + self._sent_conds.add(key) + + def _sampling_params_kwargs(self, packed, timestep) -> dict: + return { + "prompt": " ", + "guidance_scale": packed.guidance_scale, + "height": packed.height, + "width": packed.width, + "num_frames": 1, + "num_inference_steps": 1, + "save_output": False, + "suppress_logs": self.should_suppress_logs(timestep), + } + + def _execute_packed(self, packed, x, timestep): + ensure = getattr(self, "_ensure_runtime", None) + if ensure is not None: + ensure(self) + self._mark_and_maybe_drop(packed) + sampling_params = SamplingParams.from_user_sampling_params_args( + self.model_path, + server_args=self.generator.server_args, + **self._sampling_params_kwargs(packed, timestep), ) - return latents + req = prepare_request( + server_args=self.generator.server_args, + sampling_params=sampling_params, + ) + self.adapter.fill_req(req, packed) + extra = dict(req.extra or {}) + extra["comfyui_session_id"] = self.comfyui_session_id() + for key in ("comfyui_cond_key", "comfyui_cache_fp"): + value = packed.extra_req.get(key) + if value is not None: + extra[key] = value + req.extra = extra + req.generator = [ + torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt) + ] + output_batch = self.generator._send_to_scheduler_and_wait_for_response([req]) + return self.adapter.unpack(output_batch.noise_pred, packed, x) + + def forward(self, x, timestep, context, **kwargs): + packed = self.adapter.pack(x, timestep, context, **kwargs) + return self._execute_packed(packed, x, timestep) diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/flux.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/flux.py index 489d3383f5..87ecdd02bb 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/flux.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/flux.py @@ -1,69 +1,69 @@ -""" -Flux executor for SGLang Diffusion ComfyUI integration. -""" +"""Flux adapter for the ComfyUI DiT-forward contract.""" import torch -try: - from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams - from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request -except ImportError: - print( - "Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'" - ) - +from .adapter import ComfyUIModelAdapter, PackedForward from .base import SGLDiffusionExecutor +def _flux_guidance_scale(guidance) -> float: + if guidance is None: + return 3.5 + if torch.is_tensor(guidance): + return float(guidance.detach().reshape(-1)[0].item()) + return float(guidance) + + +class FluxAdapter(ComfyUIModelAdapter): + model_types = ("flux",) + pipeline_class_name = "FluxPipeline" + + def pack( + self, x, timestep, context, y=None, guidance=None, **kwargs + ) -> PackedForward: + packed = self._pack_latents(x) + t5_seq = int(context.shape[-2]) if context.ndim >= 2 else int(context.shape[0]) + clip_batch = int(y.shape[0]) if y is not None else 1 + return PackedForward( + latents=packed, + timesteps=timestep * 1000.0, + prompt_embeds=[y, context], + prompt_seq_lens=[[clip_batch], [t5_seq]], + pooled_embeds=[y], + height=x.shape[-2] * 8, + width=x.shape[-1] * 8, + guidance_scale=_flux_guidance_scale(guidance), + unpack_ctx={ + "height": x.shape[-2], + "width": x.shape[-1], + "channels": x.shape[1], + }, + ) + + def unpack(self, noise_pred, packed, x): + ctx = packed.unpack_ctx + return self._unpack_latents( + noise_pred, ctx["height"], ctx["width"], ctx["channels"] + ).to(x.device) + + @staticmethod + def _unpack_latents(latents, height, width, channels): + batch_size = latents.shape[0] + latents = latents.view(batch_size, height // 2, width // 2, channels, 2, 2) + latents = latents.permute(0, 3, 1, 4, 2, 5) + return latents.reshape(batch_size, channels, height, width) + + @staticmethod + def _pack_latents(latents): + batch_size, num_channels_latents, height, width = latents.shape + latents = latents.view( + batch_size, num_channels_latents, height // 2, 2, width // 2, 2 + ) + latents = latents.permute(0, 2, 4, 1, 3, 5) + return latents.reshape( + batch_size, (height // 2) * (width // 2), num_channels_latents * 4 + ) + + class FluxExecutor(SGLDiffusionExecutor): - """Executor for Flux models in ComfyUI.""" - - def __init__(self, generator, model_path, model, config): - super().__init__(generator, model_path, model, config) - - def forward(self, x, timestep, context, y=None, guidance=None, **kwargs): - """Forward pass for Flux model.""" - hidden_states = self._pack_latents(x) - timesteps = timestep * 1000.0 - encoder_hidden_states = context - pooled_projections = y - guidance = guidance * 1000.0 - - B, C, H, W = x.shape - height = H * 8 - width = W * 8 - # Create SamplingParams - sampling_params = SamplingParams.from_user_sampling_params_args( - self.model_path, - server_args=self.generator.server_args, - prompt=" ", - guidance_scale=3.5, # Flux typically uses embedded_cfg_scale=3.5 - height=height, - width=width, - num_frames=1, - num_inference_steps=1, - save_output=False, - suppress_logs=self.should_suppress_logs(timestep), - ) - - # Prepare request (converts SamplingParams to Req) - req = prepare_request( - server_args=self.generator.server_args, - sampling_params=sampling_params, - ) - req.latents = hidden_states # Set as [B, S, D] format directly - req.timesteps = timesteps # ComfyUI's timesteps parameter - req.prompt_embeds = [pooled_projections, encoder_hidden_states] # [CLIP, T5] - req.raw_latent_shape = torch.tensor(hidden_states.shape, dtype=torch.long) - - # Set pooled_projections (required by Flux) - req.pooled_embeds = [pooled_projections] # List format as per Req definition - req.do_classifier_free_guidance = False - req.generator = [ - torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt) - ] - - # Send request to scheduler - output_batch = self.generator._send_to_scheduler_and_wait_for_response([req]) - noise_pred = output_batch.noise_pred - return self._unpack_latents(noise_pred, H, W, C).to(x.device) + adapter_cls = FluxAdapter diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/minimax_h3.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/minimax_h3.py new file mode 100644 index 0000000000..046519ae23 --- /dev/null +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/minimax_h3.py @@ -0,0 +1,230 @@ +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-H3 adapter for the ComfyUI DiT-forward contract.""" + +from __future__ import annotations + +from dataclasses import fields +from typing import Any + +import torch + +from sglang.multimodal_gen.configs.sample.minimax_h3 import MiniMaxH3SamplingParams +from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import ( + DEFAULT_SIGMA_SHIFT_AUDIO, + DEFAULT_SIGMA_SHIFT_VIDEO, + serialize_comfyui_layout, + time_shift_sigma, + time_shift_slope, +) + +from .adapter import ComfyUIModelAdapter, PackedForward +from .base import SGLDiffusionExecutor + +# H3 pins loop/CFG fields with init=False; ComfyUI owns those. +_H3_PINNED_SAMPLING_FIELDS = frozenset( + f.name for f in fields(MiniMaxH3SamplingParams) if not f.init +) + + +def drop_h3_pinned_sampling_fields(kwargs: dict[str, Any]) -> dict[str, Any]: + for name in _H3_PINNED_SAMPLING_FIELDS: + kwargs.pop(name, None) + return kwargs + + +_WORKER_TRANSFORMER_OPTION_KEYS = ( + "minimax_h3_sigma_shift_video", + "minimax_h3_sigma_shift_audio", + "sample_sigmas", +) + + +def worker_transformer_options(opts: dict[str, Any] | None) -> dict[str, Any]: + """Keep only pickle-safe H3 keys; drop ComfyUI model_sampling etc.""" + src = opts or {} + return {key: src[key] for key in _WORKER_TRANSFORMER_OPTION_KEYS if key in src} + + +def _split_av(x) -> tuple[torch.Tensor, torch.Tensor]: + if isinstance(x, (list, tuple)) and len(x) >= 2: + return x[0], x[1] + unbind = getattr(x, "unbind", None) + if callable(unbind): + parts = unbind() + if len(parts) >= 2: + return parts[0], parts[1] + raise TypeError( + f"MiniMax H3 ComfyUI forward expects x to be [video, audio], got {type(x)!r}" + ) + + +def _as_context(context: torch.Tensor) -> torch.Tensor: + if context.ndim == 3: + return context[0] + return context + + +def _cache_fingerprint(video, sample_sigmas, serialized_layout) -> dict[str, Any]: + spatial = tuple(int(dim) for dim in video.shape[-3:]) + sigmas_fp = None + if sample_sigmas is not None: + values = ( + sample_sigmas.reshape(-1) + if torch.is_tensor(sample_sigmas) + else sample_sigmas + ) + sigmas_fp = (float(values[0]), float(values[-1]), int(len(values))) + layout_sig = None + if isinstance(serialized_layout, dict): + layout_sig = serialized_layout.get("signature") + return {"spatial": spatial, "sigmas": sigmas_fp, "layout": layout_sig} + + +class MiniMaxH3Adapter(ComfyUIModelAdapter): + model_types = ("minimax_h3",) + pipeline_class_name = "MiniMaxH3Pipeline" + + def pack(self, x, timestep, context, **kwargs) -> PackedForward: + video, audio_src = _split_av(x) + # PackedLayout is a ComfyUI class; the SGLD worker cannot unpickle it. + payload = dict(kwargs.get("minimax_payload") or {}) + layout = payload.pop("layout", None) + transformer_options = kwargs.get("transformer_options") or {} + scale = float(payload.get("audio_scale", 1.0)) + shift_v = float( + transformer_options.get( + "minimax_h3_sigma_shift_video", DEFAULT_SIGMA_SHIFT_VIDEO + ) + ) + shift_a = float( + transformer_options.get( + "minimax_h3_sigma_shift_audio", DEFAULT_SIGMA_SHIFT_AUDIO + ) + ) + sigma_v = (timestep.reshape(-1)[0] / 1000.0).to(torch.float32).clamp(min=1e-6) + sigma_a = time_shift_sigma(sigma_v, shift_v, shift_a) + carry = (sigma_a / sigma_v).to(audio_src.dtype) + audio = audio_src * carry if scale != 1.0 else audio_src + text = _as_context(context) + sample_sigmas = transformer_options.get("sample_sigmas") + extra_req: dict[str, Any] = { + "audio_latents": audio, + "h3_payload": payload, + "h3_transformer_options": worker_transformer_options(transformer_options), + "h3_sample_sigmas": sample_sigmas, + "h3_context": text, + } + serialized = serialize_comfyui_layout(layout) + if serialized is not None: + extra_req["h3_layout"] = serialized + for name in ("denoise_mask", "audio_denoise_mask"): + mask = kwargs.get(name) + if mask is not None: + extra_req[f"h3_{name}"] = mask + if sample_sigmas is not None: + extra_req["sigmas"] = sample_sigmas + extra_req["comfyui_cache_fp"] = _cache_fingerprint( + video, sample_sigmas, serialized + ) + return PackedForward( + latents=video, + timesteps=timestep, + prompt_embeds=[text], + prompt_seq_lens=[[int(text.shape[0])]], + height=int(video.shape[-2]), + width=int(video.shape[-1]), + extra_req=extra_req, + unpack_ctx={ + "audio_scale": scale, + "audio_src": audio_src, + "carry": carry, + "sigma_v": sigma_v, + "sigma_a": sigma_a, + "shift_v": shift_v, + "shift_a": shift_a, + }, + ) + + def unpack(self, noise_pred, packed, x): + video_x, audio_x = _split_av(x) + if isinstance(noise_pred, (list, tuple)): + v_video, v_audio = noise_pred[0], noise_pred[1] + else: + raise TypeError( + "MiniMax H3 unpack expects [video, audio] noise_pred, " + f"got {type(noise_pred)!r}" + ) + # Match ComfyUI MiniMaxH3._forward: negated velocities, and audio + # scaled by d(sigma_a)/d(sigma_v) so the sampler's sigma_v ODE is + # the audio stream's true ODE on its shifted schedule. + v_video = (-v_video).to(device=video_x.device, dtype=video_x.dtype) + v_audio = (-v_audio).to(device=audio_x.device, dtype=audio_x.dtype) + ctx = packed.unpack_ctx + scale = float(ctx.get("audio_scale", 1.0)) + if scale != 1.0: + audio_src = ctx["audio_src"] + carry = ctx["carry"] + sigma_a = ctx["sigma_a"] + v_audio = (1.0 - scale) * (audio_src * carry) + ( + 1.0 + (scale - 1.0) * sigma_a + ).to(v_audio.dtype) * v_audio + slope_a = time_shift_slope( + ctx["sigma_v"], + ctx.get("shift_v", DEFAULT_SIGMA_SHIFT_VIDEO), + ctx.get("shift_a", DEFAULT_SIGMA_SHIFT_AUDIO), + ).to(device=v_audio.device, dtype=v_audio.dtype) + v_audio = slope_a * v_audio + return [v_video, v_audio] + + def fill_req(self, req, packed: PackedForward) -> None: + super().fill_req(req, packed) + extra = dict(req.extra or {}) + for key in ( + "h3_payload", + "h3_transformer_options", + "h3_sample_sigmas", + "h3_context", + "h3_layout", + "h3_denoise_mask", + "h3_audio_denoise_mask", + "comfyui_cache_fp", + "comfyui_cond_key", + ): + value = packed.extra_req.get(key) + if value is not None: + extra[key] = value + req.extra = extra + + def drop_cached_fields(self, packed: PackedForward) -> None: + super().drop_cached_fields(packed) + packed.extra_req.pop("h3_payload", None) + packed.extra_req.pop("h3_transformer_options", None) + packed.extra_req.pop("h3_sample_sigmas", None) + packed.extra_req.pop("h3_context", None) + packed.extra_req.pop("h3_layout", None) + packed.extra_req.pop("h3_denoise_mask", None) + packed.extra_req.pop("h3_audio_denoise_mask", None) + packed.extra_req.pop("sigmas", None) + # Keep comfyui_cache_fp / comfyui_cond_key; later steps still send them. + + +class MiniMaxH3Executor(SGLDiffusionExecutor): + adapter_cls = MiniMaxH3Adapter + # ComfyUI MiniMaxH3.extra_conds reads this when a denoise_mask is present. + patch_size = (1, 2, 2) + + def preprocess_text_embeds(self, text_states): + """ComfyUI extra_conds runs this once per sample. + + Token refinement happens on the SGLD worker (the DiT lives there). + Identity-pass here so PackedLayout still sees the correct token length. + """ + return text_states + + def _sampling_params_kwargs(self, packed, timestep) -> dict: + kwargs = super()._sampling_params_kwargs(packed, timestep) + drop_h3_pinned_sampling_fields(kwargs) + enable_cache_dit = getattr(self, "enable_cache_dit", None) + if enable_cache_dit is not None: + kwargs["enable_cache_dit"] = bool(enable_cache_dit) + return kwargs diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/qwen_image.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/qwen_image.py index 56e1409a46..867134d9a9 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/qwen_image.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/qwen_image.py @@ -1,31 +1,40 @@ -""" -QwenImage executor for SGLang Diffusion ComfyUI integration. -""" - -import torch - -try: - from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams - from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request -except ImportError: - print( - "Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'" - ) +"""Qwen-Image adapters for the ComfyUI DiT-forward contract.""" import comfy.ldm.common_dit +from .adapter import ComfyUIModelAdapter, PackedForward from .base import SGLDiffusionExecutor -class QwenImageExecutor(SGLDiffusionExecutor): - """Executor for QwenImage models in ComfyUI.""" +class QwenImageAdapter(ComfyUIModelAdapter): + model_types = ("qwen_image",) + pipeline_class_name = "QwenImagePipeline" + patch_size = 2 - def __init__(self, generator, model_path, model, config): - super().__init__(generator, model_path, model, config) - self.patch_size = 2 + def pack(self, x, timestep, context, **kwargs) -> PackedForward: + latents, orig_shape = self._pack_latents(x) + seq = int(context.shape[-2]) if context.ndim >= 2 else int(context.shape[0]) + return PackedForward( + latents=latents, + timesteps=timestep * 1000.0, + prompt_embeds=[context], + prompt_seq_lens=[[seq]], + height=orig_shape[-2] * 8, + width=orig_shape[-1] * 8, + unpack_ctx={ + "num_embeds": latents.shape[1], + "orig_shape": orig_shape, + "x": x, + }, + ) + + def unpack(self, noise_pred, packed, x): + ctx = packed.unpack_ctx + return self._unpack_latents( + noise_pred, ctx["num_embeds"], ctx["orig_shape"], ctx["x"] + ) def _pack_latents(self, x): - """Process hidden states for QwenImage model.""" latents = comfy.ldm.common_dit.pad_to_patch_size( x, (1, self.patch_size, self.patch_size) ) @@ -47,8 +56,8 @@ class QwenImageExecutor(SGLDiffusionExecutor): ) return latents, orig_shape - def _unpack_latents(self, latents, num_embeds, orig_shape, x): - """Unpack hidden states from packed format to standard format.""" + @staticmethod + def _unpack_latents(latents, num_embeds, orig_shape, x): latents = latents[:, :num_embeds].view( orig_shape[0], orig_shape[-3], @@ -59,57 +68,14 @@ class QwenImageExecutor(SGLDiffusionExecutor): 2, ) latents = latents.permute(0, 4, 1, 2, 5, 3, 6) - latents = latents.reshape(orig_shape)[:, :, :, : x.shape[-2], : x.shape[-1]] - return latents - - def forward(self, x, timestep, context, **kwargs): - """Forward pass for QwenImage model.""" - latents, orig_shape = self._pack_latents(x) - num_embeds = latents.shape[1] - height = orig_shape[-2] * 8 - width = orig_shape[-1] * 8 - - sampling_params = SamplingParams.from_user_sampling_params_args( - self.model_path, - server_args=self.generator.server_args, - prompt=" ", - guidance_scale=1.0, - height=height, - width=width, - num_frames=1, - num_inference_steps=1, - save_output=False, - suppress_logs=self.should_suppress_logs(timestep), - ) - - # Prepare request (converts SamplingParams to Req) - req = prepare_request( - server_args=self.generator.server_args, - sampling_params=sampling_params, - ) - # Set ComfyUI-specific inputs directly on the Req object - req.latents = latents - req.timesteps = timestep * 1000.0 - req.prompt_embeds = [context] - req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long) - req.do_classifier_free_guidance = False - req.generator = [ - torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt) - ] - - output_batch = self.generator._send_to_scheduler_and_wait_for_response([req]) - noise_pred = output_batch.noise_pred - - return self._unpack_latents(noise_pred, num_embeds, orig_shape, x) + return latents.reshape(orig_shape)[:, :, :, : x.shape[-2], : x.shape[-1]] -class QwenImageEditExecutor(QwenImageExecutor): - """Executor for QwenImageEdit models in ComfyUI.""" +class QwenImageEditAdapter(QwenImageAdapter): + model_types = ("qwen_image_edit",) + pipeline_class_name = "QwenImageEditPlusPipeline" - def __init__(self, generator, model_path, model, config): - super().__init__(generator, model_path, model, config) - - def forward( + def pack( self, x, timestep, @@ -117,56 +83,22 @@ class QwenImageEditExecutor(QwenImageExecutor): attention_mask=None, ref_latents=None, additional_t_cond=None, - transformer_options={}, + transformer_options=None, **kwargs, - ): - """Forward pass for QwenImageEdit model.""" - latents, orig_shape = self._pack_latents(x) - num_embeds = latents.shape[1] - height = orig_shape[-2] * 8 - width = orig_shape[-1] * 8 + ) -> PackedForward: + packed = super().pack(x, timestep, context, **kwargs) + if ref_latents: + pack_ref, orig_ref_shape = self._pack_latents(ref_latents[0]) + packed.extra_req["image_latent"] = pack_ref + packed.extra_req["vae_image_sizes"] = [ + (orig_ref_shape[-1], orig_ref_shape[-2]) + ] + return packed - # Prepare vae_image_sizes for the condition image (ref_latents) - vae_image_sizes = [] - pack_ref_latents = None - # TODO: sgld now don't support multiple condition images, so we only support one condition image for now. - if ref_latents is not None and len(ref_latents) > 0: - pack_ref_latents, orig_ref_shape = self._pack_latents(ref_latents[0]) - vae_image_sizes = [(orig_ref_shape[-1], orig_ref_shape[-2])] +class QwenImageExecutor(SGLDiffusionExecutor): + adapter_cls = QwenImageAdapter - sampling_params = SamplingParams.from_user_sampling_params_args( - self.model_path, - server_args=self.generator.server_args, - prompt=" ", - guidance_scale=1.0, - image_path="", - height=height, - width=width, - num_frames=1, - num_inference_steps=1, - save_output=False, - suppress_logs=self.should_suppress_logs(timestep), - ) - # Prepare request (converts SamplingParams to Req) - req = prepare_request( - server_args=self.generator.server_args, - sampling_params=sampling_params, - ) - # Set ComfyUI-specific inputs directly on the Req object - req.latents = latents - req.image_latent = pack_ref_latents - req.timesteps = timestep * 1000.0 - req.vae_image_sizes = vae_image_sizes - req.prompt_embeds = [context] - req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long) - req.do_classifier_free_guidance = False - req.generator = [ - torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt) - ] - - output_batch = self.generator._send_to_scheduler_and_wait_for_response([req]) - noise_pred = output_batch.noise_pred - - return self._unpack_latents(noise_pred, num_embeds, orig_shape, x) +class QwenImageEditExecutor(SGLDiffusionExecutor): + adapter_cls = QwenImageEditAdapter diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/zimage.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/zimage.py index d817c4b1a2..f1fe3f8038 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/zimage.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/executors/zimage.py @@ -1,64 +1,30 @@ -""" -ZImage executor for SGLang Diffusion ComfyUI integration. -""" - -import torch - -try: - from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams - from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request -except ImportError: - print( - "Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'" - ) +"""Z-Image (lumina2) adapter for the ComfyUI DiT-forward contract.""" +from .adapter import ComfyUIModelAdapter, PackedForward from .base import SGLDiffusionExecutor -class ZImageExecutor(SGLDiffusionExecutor): - """Executor for ZImage models in ComfyUI.""" +class ZImageAdapter(ComfyUIModelAdapter): + model_types = ("lumina2",) + pipeline_class_name = "ZImagePipeline" - def __init__(self, generator, model_path, model, config): - super().__init__(generator, model_path, model, config) - - def forward(self, x, timesteps, context, **kwargs): - """Forward pass for ZImage model.""" - B, C, H, W = x.shape - height = H * 8 - width = W * 8 - sampling_params = SamplingParams.from_user_sampling_params_args( - self.model_path, - server_args=self.generator.server_args, - prompt=" ", - guidance_scale=1.0, - height=height, - width=width, - num_frames=1, # For images - num_inference_steps=1, # Single step for ComfyUI - save_output=False, - suppress_logs=self.should_suppress_logs(timesteps), - ) - - # Prepare request (converts SamplingParams to Req) - req = prepare_request( - server_args=self.generator.server_args, - sampling_params=sampling_params, - ) - latents = x.unsqueeze(2) + def pack(self, x, timestep, context, **kwargs) -> PackedForward: context = context.squeeze(0) - # Set ComfyUI-specific inputs directly on the Req object - req.latents = latents # ComfyUI's x parameter - req.timesteps = timesteps * 1000.0 # ComfyUI's timesteps parameter - req.prompt_embeds = [ - context - ] # ComfyUI's context parameter (must be List[Tensor]) - req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long) - req.do_classifier_free_guidance = False - req.generator = [ - torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt) - ] + return PackedForward( + latents=x.unsqueeze(2), + timesteps=timestep * 1000.0, + prompt_embeds=[context], + prompt_seq_lens=[[int(context.shape[0])]], + height=x.shape[-2] * 8, + width=x.shape[-1] * 8, + ) - output_batch = self.generator._send_to_scheduler_and_wait_for_response([req]) - noise_pred = output_batch.noise_pred + def unpack(self, noise_pred, packed, x): + # SGLD returns 5D [B, C, T, H, W]; ComfyUI samples 4D [B, C, H, W]. + if noise_pred.ndim == 5: + noise_pred = noise_pred.squeeze(2) + return noise_pred.to(x.device) - return noise_pred.permute(1, 0, 2, 3).to(x.device) + +class ZImageExecutor(SGLDiffusionExecutor): + adapter_cls = ZImageAdapter diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/nodes.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/nodes.py index f3d2275ee1..ac0c7d9878 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/nodes.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/nodes.py @@ -10,6 +10,16 @@ import folder_paths import torch from .core import SGLDiffusionGenerator, SGLDiffusionServerAPI + + +def _enable_gguf_in_diffusion_models() -> None: + """Let SGLDUNETLoader list ``.gguf`` DiTs. Quantization lives in the file.""" + entry = folder_paths.folder_names_and_paths.get("diffusion_models") + if entry is not None and len(entry) >= 2 and isinstance(entry[1], set): + entry[1].add(".gguf") + + +_enable_gguf_in_diffusion_models() from .utils import ( convert_b64_to_tensor_image, convert_video_to_comfy_video, @@ -25,7 +35,14 @@ class SGLDOptions: "required": {}, "optional": { "model_type": ( - ["auto-detect", "qwen_image", "qwen_image_edit", "flux", "lumina2"], + [ + "auto-detect", + "qwen_image", + "qwen_image_edit", + "flux", + "lumina2", + "minimax_h3", + ], {"default": "auto-detect"}, ), "enable_torch_compile": ( @@ -61,6 +78,25 @@ class SGLDOptions: "STRING", {"default": ""}, ), + "dit_layerwise_offload": ( + "BOOLEAN", + {"default": False}, + ), + "enable_cache_dit": ( + "BOOLEAN", + {"default": False}, + ), + "quantization": ( + "STRING", + {"default": ""}, + ), + "transformer_weights_path": ( + "STRING", + { + "default": "", + "multiline": False, + }, + ), }, } @@ -82,6 +118,10 @@ class SGLDOptions: dp_degree: int = 1, enable_cfg_parallel: bool = False, attention_backend: str = "", + dit_layerwise_offload: bool = False, + enable_cache_dit: bool = False, + quantization: str = "", + transformer_weights_path: str = "", ): """ Build a dictionary of SGLang Diffusion runtime options. @@ -89,7 +129,12 @@ class SGLDOptions: # Convert -1 to None for optional parameters (matching ServerArgs defaults) ulysses_degree = None if ulysses_degree == -1 else ulysses_degree ring_degree = None if ring_degree == -1 else ring_degree + tp_size = None if tp_size == -1 else tp_size + sp_degree = None if sp_degree == -1 else sp_degree attention_backend = None if attention_backend == "" else attention_backend + # dp_degree is a leftover alias; ServerArgs only has dp_size. + if dp_degree not in (None, 1) and dp_size in (None, 1): + dp_size = dp_degree options = { "model_type": model_type, @@ -100,10 +145,20 @@ class SGLDOptions: "ulysses_degree": ulysses_degree, "ring_degree": ring_degree, "dp_size": dp_size, - "dp_degree": dp_degree, "enable_cfg_parallel": enable_cfg_parallel, "attention_backend": attention_backend, + "dit_layerwise_offload": dit_layerwise_offload, } + if enable_cache_dit: + options["enable_cache_dit"] = True + quantization = (quantization or "").strip() + if quantization: + options["quantization"] = quantization + transformer_weights_path = (transformer_weights_path or "").strip() + if transformer_weights_path: + # Same selector as `sglang serve --transformer-weights-path`: + # local .gguf, owner/repo/path/file.gguf, or owner/repo:QUANT. + options["transformer_weights_path"] = transformer_weights_path # Strip None to keep payload clean options = {k: v for k, v in options.items() if v is not None} @@ -165,7 +220,7 @@ class SGLDLoraLoader: class SGLDUNETLoader: def __init__(self): - self.generator = SGLDiffusionGenerator() + self.generator = SGLDiffusionGenerator.shared() @classmethod def INPUT_TYPES(s): diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/README.md b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/README.md index 5246d29231..ecf099faf4 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/README.md +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/README.md @@ -1,13 +1,16 @@ # ComfyUI SGLDiffusion Pipeline Tests -This directory contains tests for each ComfyUI pipeline integration. +These e2e tests start the **native** Flux / Qwen-Image / Z-Image pipeline with +`comfyui_mode=True` and feed pre-encoded latents, timesteps, and embeddings the +way the plugin adapter would. Adapter, session, and CUDA IPC unit tests live +under `python/sglang/multimodal_gen/test/unit/`. ## Test Files -- `test_zimage_pipeline.py` - Tests for ComfyUIZImagePipeline -- `test_flux_pipeline.py` - Tests for ComfyUIFluxPipeline -- `test_qwen_image_pipeline.py` - Tests for ComfyUIQwenImagePipeline -- `test_qwen_image_edit_pipeline.py` - Tests for ComfyUIQwenImageEditPipeline (I2I/edit mode) +- `test_zimage_pipeline.py` - Z-Image native pipeline under `--comfyui-mode` +- `test_flux_pipeline.py` - Flux native pipeline under `--comfyui-mode` +- `test_qwen_image_pipeline.py` - Qwen-Image native pipeline under `--comfyui-mode` +- `test_qwen_image_edit_pipeline.py` - Qwen-Image-Edit native pipeline (I2I/edit; experimental) ## Running Tests diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_flux_pipeline.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_flux_pipeline.py index 59147a6c39..7c3c02e361 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_flux_pipeline.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_flux_pipeline.py @@ -1,4 +1,4 @@ -"""Test for ComfyUIFluxPipeline with pass-through scheduler.""" +"""Test for FluxPipeline with pass-through scheduler.""" import os import sys @@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request def test_comfyui_flux_pipeline_direct() -> None: - """Test ComfyUIFluxPipeline with custom inputs.""" + """Test FluxPipeline with custom inputs.""" model_path = os.environ.get( "SGLANG_TEST_FLUX_MODEL_PATH", "black-forest-labs/FLUX.1-dev", # Supports both safetensors file and diffusers format @@ -20,7 +20,8 @@ def test_comfyui_flux_pipeline_direct() -> None: generator = DiffGenerator.from_pretrained( model_path=model_path, - pipeline_class_name="ComfyUIFluxPipeline", + model_id="FLUX.1-dev", + pipeline_class_name="FluxPipeline", num_gpus=2, comfyui_mode=True, ) @@ -68,6 +69,7 @@ def test_comfyui_flux_pipeline_direct() -> None: width=width, num_frames=1, num_inference_steps=1, + guidance_scale=1.0, save_output=True, return_trajectory_latents=True, ) @@ -83,6 +85,10 @@ def test_comfyui_flux_pipeline_direct() -> None: clip_dim = 768 req.prompt_embeds = [pooled_projections, encoder_hidden_states] + req.prompt_seq_lens = [ + [int(pooled_projections.shape[0])], + [encoder_seq_len], + ] if req.guidance_scale > 1.0: dummy_neg_clip_embedding = torch.zeros( @@ -103,11 +109,15 @@ def test_comfyui_flux_pipeline_direct() -> None: dummy_neg_clip_embedding, negative_encoder_hidden_states, ] + req.negative_prompt_seq_lens = [ + [int(dummy_neg_clip_embedding.shape[0])], + [encoder_seq_len], + ] else: req.negative_prompt_embeds = None req.pooled_embeds = [pooled_projections] - req.neg_pooled_embeds = [] + req.neg_pooled_embeds = [torch.zeros_like(pooled_projections)] if ( req.guidance_scale > 1.0 diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_qwen_image_edit_pipeline.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_qwen_image_edit_pipeline.py index 422441d568..017f3c3fab 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_qwen_image_edit_pipeline.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_qwen_image_edit_pipeline.py @@ -1,4 +1,4 @@ -"""Test for ComfyUIQwenImageEditPipeline with pass-through scheduler (I2I/edit mode).""" +"""Test for QwenImageEditPlusPipeline with pass-through scheduler (I2I/edit mode).""" import os import sys @@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request def test_comfyui_qwen_image_edit_pipeline_direct() -> None: - """Test ComfyUIQwenImageEditPipeline with edit mode (I2I) and custom inputs.""" + """Test QwenImageEditPlusPipeline with edit mode (I2I) and custom inputs.""" model_path = os.environ.get( "SGLANG_TEST_QWEN_IMAGE_EDIT_MODEL_PATH", "Qwen/Qwen-Image-Edit-2511", # Supports both safetensors file and diffusers format @@ -20,7 +20,7 @@ def test_comfyui_qwen_image_edit_pipeline_direct() -> None: generator = DiffGenerator.from_pretrained( model_path=model_path, - pipeline_class_name="ComfyUIQwenImageEditPipeline", + pipeline_class_name="QwenImageEditPlusPipeline", num_gpus=1, comfyui_mode=True, dit_layerwise_offload=False, diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_qwen_image_pipeline.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_qwen_image_pipeline.py index bb7b070f07..671d731472 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_qwen_image_pipeline.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_qwen_image_pipeline.py @@ -1,4 +1,4 @@ -"""Test for ComfyUIQwenImagePipeline with pass-through scheduler.""" +"""Test for QwenImagePipeline with pass-through scheduler.""" import os import sys @@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request def test_comfyui_qwen_image_pipeline_direct() -> None: - """Test ComfyUIQwenImagePipeline with custom inputs.""" + """Test QwenImagePipeline with custom inputs.""" model_path = os.environ.get( "SGLANG_TEST_QWEN_IMAGE_MODEL_PATH", "Qwen/Qwen-Image", # Supports both safetensors file and diffusers format @@ -20,7 +20,7 @@ def test_comfyui_qwen_image_pipeline_direct() -> None: generator = DiffGenerator.from_pretrained( model_path=model_path, - pipeline_class_name="ComfyUIQwenImagePipeline", + pipeline_class_name="QwenImagePipeline", num_gpus=2, comfyui_mode=True, dit_layerwise_offload=False, diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_zimage_pipeline.py b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_zimage_pipeline.py index 4b053f1dbc..93f1112e32 100644 --- a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_zimage_pipeline.py +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/test/test_zimage_pipeline.py @@ -1,4 +1,4 @@ -"""Test for ComfyUIZImagePipeline with pass-through scheduler.""" +"""Test for ZImagePipeline with pass-through scheduler.""" import os import sys @@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request def test_comfyui_zimage_pipeline_direct() -> None: - """Test ComfyUIZImagePipeline with custom inputs.""" + """Test ZImagePipeline with custom inputs.""" model_path = os.environ.get( "SGLANG_TEST_ZIMAGE_MODEL_PATH", "Tongyi-MAI/Z-Image-Turbo", # Supports both safetensors file and diffusers format @@ -20,7 +20,7 @@ def test_comfyui_zimage_pipeline_direct() -> None: generator = DiffGenerator.from_pretrained( model_path=model_path, - pipeline_class_name="ComfyUIZImagePipeline", + pipeline_class_name="ZImagePipeline", num_gpus=1, sp_degree=1, comfyui_mode=True, @@ -77,6 +77,7 @@ def test_comfyui_zimage_pipeline_direct() -> None: req.latents = latents req.timesteps = timesteps req.prompt_embeds = [context] + req.prompt_seq_lens = [[context_seq_len]] req.negative_prompt_embeds = None req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long) diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_r2v_sgld.json b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_r2v_sgld.json new file mode 100644 index 0000000000..c5ee0c712e --- /dev/null +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_r2v_sgld.json @@ -0,0 +1,136 @@ +{ + "6": { + "class_type": "SGLDUNETLoader", + "inputs": { + "unet_name": "minimax_h3_ref2va_bf16.safetensors", + "weight_dtype": "default", + "sgld_options": ["52", 0] + } + }, + "52": { + "class_type": "SGLDOptions", + "inputs": { + "model_type": "minimax_h3", + "enable_torch_compile": false, + "num_gpus": 1, + "tp_size": -1, + "sp_degree": -1, + "ulysses_degree": -1, + "ring_degree": -1, + "dp_size": 1, + "dp_degree": 1, + "enable_cfg_parallel": false, + "attention_backend": "" + } + }, + "13": { + "class_type": "CLIPLoader", + "inputs": { + "clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", + "type": "minimax", + "device": "default" + } + }, + "11": { + "class_type": "VAELoader", + "inputs": { + "vae_name": "minimax_h3_video_vae_fp16.safetensors" + } + }, + "24": { + "class_type": "VAELoader", + "inputs": { + "vae_name": "minimax_h3_audio_vae_fp32.safetensors" + } + }, + "137": { + "class_type": "LoadImage", + "inputs": { + "image": "sglang_girl.jpg" + } + }, + "136": { + "class_type": "MiniMaxH3ReferenceToVideo", + "inputs": { + "clip": ["13", 0], + "vae": ["11", 0], + "audio_vae": ["24", 0], + "prompt": "Use as the sole character and style reference. A young woman walks slowly through a quiet garden at golden hour, looking toward the camera, then turning to watch leaves drift past. Soft cinematic lighting, shallow depth of field, gentle breeze in her hair.\n\nAudio: quiet outdoor ambience, faint birds, a soft piano motif, no lyrics.", + "width": 864, + "height": 480, + "length": 73, + "ref_image_size": "match", + "ref_images.ref_image_0": ["137", 0] + } + }, + "17": { + "class_type": "KSamplerSelect", + "inputs": { + "sampler_name": "res_multistep" + } + }, + "9": { + "class_type": "BasicScheduler", + "inputs": { + "model": ["6", 0], + "scheduler": "beta", + "steps": 8, + "denoise": 1.0 + } + }, + "15": { + "class_type": "RandomNoise", + "inputs": { + "noise_seed": 1101 + } + }, + "16": { + "class_type": "BasicGuider", + "inputs": { + "model": ["6", 0], + "conditioning": ["136", 0] + } + }, + "14": { + "class_type": "SamplerCustomAdvanced", + "inputs": { + "noise": ["15", 0], + "guider": ["16", 0], + "sampler": ["17", 0], + "sigmas": ["9", 0], + "latent_image": ["136", 1] + } + }, + "10": { + "class_type": "VAEDecode", + "inputs": { + "samples": ["14", 0], + "vae": ["11", 0] + } + }, + "23": { + "class_type": "VAEDecodeAudio", + "inputs": { + "samples": ["14", 0], + "vae": ["24", 0] + } + }, + "91": { + "class_type": "CreateVideo", + "inputs": { + "images": ["10", 0], + "audio": ["23", 0], + "fps": 24, + "bit_depth": 8 + } + }, + "92": { + "class_type": "SaveVideo", + "inputs": { + "video": ["91", 0], + "filename_prefix": "video/MiniMax_H3_sgld_r2v", + "format": "auto", + "codec": "auto" + } + } +} diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_t2v_sgld.json b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_t2v_sgld.json new file mode 100644 index 0000000000..d6252320d4 --- /dev/null +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_t2v_sgld.json @@ -0,0 +1,133 @@ +{ + "6": { + "class_type": "SGLDUNETLoader", + "inputs": { + "unet_name": "minimax_h3_fl2va_bf16.safetensors", + "weight_dtype": "default", + "sgld_options": ["52", 0] + }, + "_meta": { + "title": "SGLDiffusion UNET Loader" + } + }, + "52": { + "class_type": "SGLDOptions", + "inputs": { + "model_type": "minimax_h3", + "enable_torch_compile": false, + "num_gpus": 1, + "tp_size": -1, + "sp_degree": -1, + "ulysses_degree": -1, + "ring_degree": -1, + "dp_size": 1, + "dp_degree": 1, + "enable_cfg_parallel": false, + "attention_backend": "" + }, + "_meta": { + "title": "SGLDiffusion Options" + } + }, + "13": { + "class_type": "CLIPLoader", + "inputs": { + "clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", + "type": "minimax", + "device": "default" + } + }, + "11": { + "class_type": "VAELoader", + "inputs": { + "vae_name": "minimax_h3_video_vae_fp16.safetensors" + } + }, + "24": { + "class_type": "VAELoader", + "inputs": { + "vae_name": "minimax_h3_audio_vae_fp32.safetensors" + } + }, + "104": { + "class_type": "MiniMaxH3ImageToVideo", + "inputs": { + "clip": ["13", 0], + "vae": ["11", 0], + "prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.", + "width": 864, + "height": 480, + "length": 73 + } + }, + "17": { + "class_type": "KSamplerSelect", + "inputs": { + "sampler_name": "res_multistep" + } + }, + "9": { + "class_type": "BasicScheduler", + "inputs": { + "model": ["6", 0], + "scheduler": "simple", + "steps": 8, + "denoise": 1.0 + } + }, + "15": { + "class_type": "RandomNoise", + "inputs": { + "noise_seed": 1101 + } + }, + "16": { + "class_type": "BasicGuider", + "inputs": { + "model": ["6", 0], + "conditioning": ["104", 0] + } + }, + "14": { + "class_type": "SamplerCustomAdvanced", + "inputs": { + "noise": ["15", 0], + "guider": ["16", 0], + "sampler": ["17", 0], + "sigmas": ["9", 0], + "latent_image": ["104", 1] + } + }, + "10": { + "class_type": "VAEDecode", + "inputs": { + "samples": ["14", 0], + "vae": ["11", 0] + } + }, + "23": { + "class_type": "VAEDecodeAudio", + "inputs": { + "samples": ["14", 0], + "vae": ["24", 0] + } + }, + "91": { + "class_type": "CreateVideo", + "inputs": { + "images": ["10", 0], + "audio": ["23", 0], + "fps": 24, + "bit_depth": 8 + } + }, + "92": { + "class_type": "SaveVideo", + "inputs": { + "video": ["91", 0], + "filename_prefix": "video/MiniMax_H3_sgld_t2v", + "format": "auto", + "codec": "auto" + } + } +} diff --git a/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_t2v_sgld_upscaler.json b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_t2v_sgld_upscaler.json new file mode 100644 index 0000000000..dbeae783f6 --- /dev/null +++ b/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/minimax_h3_t2v_sgld_upscaler.json @@ -0,0 +1,241 @@ +{ + "6": { + "class_type": "SGLDUNETLoader", + "inputs": { + "unet_name": "minimax_h3_fl2va_bf16.safetensors", + "weight_dtype": "default", + "sgld_options": ["52", 0] + } + }, + "7": { + "class_type": "SGLDLoraLoader", + "inputs": { + "model": ["6", 0], + "lora_name": "minimax_h3_fl2v_lightx2v_turbo_4step_v0.1_comfy.safetensors", + "strength_model": 1.0, + "nickname": "lightx2v", + "target": "all" + } + }, + "52": { + "class_type": "SGLDOptions", + "inputs": { + "model_type": "minimax_h3", + "enable_torch_compile": false, + "num_gpus": 1, + "tp_size": -1, + "sp_degree": -1, + "ulysses_degree": -1, + "ring_degree": -1, + "dp_size": 1, + "dp_degree": 1, + "enable_cfg_parallel": false, + "attention_backend": "", + "dit_layerwise_offload": false, + "enable_cache_dit": false, + "quantization": "", + "transformer_weights_path": "" + } + }, + "13": { + "class_type": "CLIPLoader", + "inputs": { + "clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", + "type": "minimax", + "device": "default" + } + }, + "11": { + "class_type": "VAELoader", + "inputs": { + "vae_name": "minimax_h3_video_vae_fp16.safetensors" + } + }, + "24": { + "class_type": "VAELoader", + "inputs": { + "vae_name": "minimax_h3_audio_vae_fp32.safetensors" + } + }, + "104": { + "class_type": "MiniMaxH3ImageToVideo", + "inputs": { + "clip": ["13", 0], + "vae": ["11", 0], + "prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.", + "width": 768, + "height": 448, + "length": 39 + } + }, + "17": { + "class_type": "KSamplerSelect", + "inputs": { + "sampler_name": "res_multistep" + } + }, + "9": { + "class_type": "BasicScheduler", + "inputs": { + "model": ["7", 0], + "scheduler": "simple", + "steps": 4, + "denoise": 1.0 + } + }, + "15": { + "class_type": "RandomNoise", + "inputs": { + "noise_seed": 7703 + } + }, + "16": { + "class_type": "BasicGuider", + "inputs": { + "model": ["7", 0], + "conditioning": ["104", 0] + } + }, + "14": { + "class_type": "SamplerCustomAdvanced", + "inputs": { + "noise": ["15", 0], + "guider": ["16", 0], + "sampler": ["17", 0], + "sigmas": ["9", 0], + "latent_image": ["104", 1] + } + }, + "93": { + "class_type": "VAEDecode", + "inputs": { + "samples": ["14", 0], + "vae": ["11", 0] + } + }, + "23": { + "class_type": "VAEDecodeAudio", + "inputs": { + "samples": ["14", 0], + "vae": ["24", 0] + } + }, + "94": { + "class_type": "CreateVideo", + "inputs": { + "images": ["93", 0], + "audio": ["23", 0], + "fps": 24, + "bit_depth": 8 + } + }, + "95": { + "class_type": "SaveVideo", + "inputs": { + "video": ["94", 0], + "filename_prefix": "video/MiniMax_H3_sgld_t2v_upscaler_low", + "format": "auto", + "codec": "auto" + } + }, + "201": { + "class_type": "LTXVSeparateAVLatent", + "inputs": { + "av_latent": ["14", 0] + } + }, + "202": { + "class_type": "MinimaxH3LatentUpscaler3D", + "inputs": { + "latent": ["201", 0], + "model_name": "minimax_h3_latent_upscaler_3d_fp16.safetensors", + "mode": "scale by multiplier", + "mode.scale": 2.0, + "align": 32, + "device": "cuda", + "precision": "fp16" + } + }, + "203": { + "class_type": "LTXVConcatAVLatent", + "inputs": { + "video_latent": ["202", 0], + "audio_latent": ["201", 1] + } + }, + "204": { + "class_type": "MiniMaxH3ImageToVideo", + "inputs": { + "clip": ["13", 0], + "vae": ["11", 0], + "prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.", + "width": 1536, + "height": 896, + "length": 39 + } + }, + "209": { + "class_type": "BasicScheduler", + "inputs": { + "model": ["7", 0], + "scheduler": "simple", + "steps": 4, + "denoise": 0.45 + } + }, + "215": { + "class_type": "RandomNoise", + "inputs": { + "noise_seed": 7704 + } + }, + "216": { + "class_type": "BasicGuider", + "inputs": { + "model": ["7", 0], + "conditioning": ["204", 0] + } + }, + "214": { + "class_type": "SamplerCustomAdvanced", + "inputs": { + "noise": ["215", 0], + "guider": ["216", 0], + "sampler": ["17", 0], + "sigmas": ["209", 0], + "latent_image": ["203", 0] + } + }, + "10": { + "class_type": "VAEDecode", + "inputs": { + "samples": ["214", 0], + "vae": ["11", 0] + } + }, + "96": { + "class_type": "VAEDecodeAudio", + "inputs": { + "samples": ["214", 0], + "vae": ["24", 0] + } + }, + "91": { + "class_type": "CreateVideo", + "inputs": { + "images": ["10", 0], + "audio": ["96", 0], + "fps": 24, + "bit_depth": 8 + } + }, + "92": { + "class_type": "SaveVideo", + "inputs": { + "video": ["91", 0], + "filename_prefix": "video/MiniMax_H3_sgld_t2v_upscaler", + "format": "auto", + "codec": "auto" + } + } +} diff --git a/python/sglang/multimodal_gen/configs/models/dits/minimax_h3.py b/python/sglang/multimodal_gen/configs/models/dits/minimax_h3.py index efd5484dfd..9cd41febb2 100644 --- a/python/sglang/multimodal_gen/configs/models/dits/minimax_h3.py +++ b/python/sglang/multimodal_gen/configs/models/dits/minimax_h3.py @@ -104,6 +104,7 @@ class MiniMaxH3DiTArchConfig(DiTArchConfig): norm_eps: float = 1e-5 qk_norm_eps: float = 1e-5 final_norm_eps: float = 1e-5 + qkv_checkpoint_grouped: bool = True checkpoint_uses_diffusers_layout: bool = False adaln_affine_input_dim: int | None = None has_gate_compress: bool = False diff --git a/python/sglang/multimodal_gen/configs/models/vaes/minimax_h3_audio.py b/python/sglang/multimodal_gen/configs/models/vaes/minimax_h3_audio.py index b3847a44bd..6504ee8481 100644 --- a/python/sglang/multimodal_gen/configs/models/vaes/minimax_h3_audio.py +++ b/python/sglang/multimodal_gen/configs/models/vaes/minimax_h3_audio.py @@ -33,6 +33,11 @@ class MiniMaxH3AudioVAEConfig(VAEConfig): super().update_model_arch(model_dict) def post_init(self) -> None: + if ( + self.arch_config.latents_mean is None + and self.arch_config.latents_std is None + ): + return validate_minimax_h3_vae_latent_stats( self.arch_config, component_name="audio_vae", diff --git a/python/sglang/multimodal_gen/configs/models/vaes/minimax_h3_video.py b/python/sglang/multimodal_gen/configs/models/vaes/minimax_h3_video.py index 4fcc6540f5..0a790a7377 100644 --- a/python/sglang/multimodal_gen/configs/models/vaes/minimax_h3_video.py +++ b/python/sglang/multimodal_gen/configs/models/vaes/minimax_h3_video.py @@ -69,6 +69,12 @@ class MiniMaxH3VideoVAEConfig(VAEConfig): def post_init(self) -> None: self.resolved_parallel_decode_mode() + if ( + self.arch_config.latents_mean is None + and self.arch_config.latents_std is None + ): + # ComfyUI DiT-forward never loads this VAE; stats stay unset. + return validate_minimax_h3_vae_latent_stats( self.arch_config, component_name="video_vae", diff --git a/python/sglang/multimodal_gen/runtime/disaggregation/scheduler_mixin.py b/python/sglang/multimodal_gen/runtime/disaggregation/scheduler_mixin.py index 633893b876..2001f1fb56 100644 --- a/python/sglang/multimodal_gen/runtime/disaggregation/scheduler_mixin.py +++ b/python/sglang/multimodal_gen/runtime/disaggregation/scheduler_mixin.py @@ -54,6 +54,10 @@ from sglang.multimodal_gen.runtime.disaggregation.transport.protocol import ( encode_transfer_msg, is_transfer_message, ) +from sglang.multimodal_gen.runtime.distributed.ipc_cuda import ( + attach_cuda_tensors, + detach_cuda_tensors, +) from sglang.multimodal_gen.runtime.distributed.utils import broadcast_pyobj from sglang.multimodal_gen.runtime.entrypoints.utils import expand_request_outputs from sglang.multimodal_gen.runtime.pipelines_core import Req @@ -793,6 +797,31 @@ class SchedulerDisaggMixin: return data + def _broadcast_recv_reqs(self: Scheduler, recv_reqs): + """ComfyUI multi-rank recv: pickle the Req skeleton, NCCL the CUDA tensors. + + The general SP/CFG/TP path stays in ``Scheduler.recv_reqs`` as the + original whole-list ``broadcast_pyobj``. This helper is only the + ComfyUI overlay and does not use disagg extract. + """ + is_rank0 = self.gpu_id == 0 + if is_rank0: + assert recv_reqs is not None, "rank 0 must pass the ZMQ poll result" + skeleton, tensors = detach_cuda_tensors(recv_reqs) + else: + skeleton, tensors = None, None + + skeleton = self._broadcast_to_all_ranks(skeleton) + tensors = self._broadcast_tensor_dict_to_all_ranks(tensors) + if is_rank0: + return recv_reqs + if not skeleton: + return [] + local_device = torch.device( + f"{current_platform.device_type}:{self.worker.local_rank}" + ) + return attach_cuda_tensors(skeleton, tensors or {}, device=local_device) + def _is_multi_rank(self: Scheduler) -> bool: sa = self.server_args return sa.sp_degree != 1 or sa.tp_size > 1 or sa.enable_cfg_parallel diff --git a/python/sglang/multimodal_gen/runtime/distributed/ipc_cuda.py b/python/sglang/multimodal_gen/runtime/distributed/ipc_cuda.py new file mode 100644 index 0000000000..8a9979497f --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/distributed/ipc_cuda.py @@ -0,0 +1,288 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Keep CUDA tensors off the pickle path for local scheduler hops. + +Two hops share the same tree walk: + +- ComfyUI / DiffGenerator ↔ rank-0: replace tensors with ``CudaIpcRef`` + handles before pickle, rebuild with ``UntypedStorage._new_shared_cuda`` +- ComfyUI multi-rank recv: detach tensors for NCCL, pickle the skeleton, + attach them again (not the disagg extract path) +""" + +from __future__ import annotations + +import copy +import dataclasses +from collections import OrderedDict +from dataclasses import dataclass +from typing import Any + +import torch + +from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger + +logger = init_logger(__name__) + +# Producer-side retain: CUDA IPC cannot open a handle in the same process, +# and the allocation must stay alive until the consumer maps it. +# Do not LRU-evict: a live handle that is dropped before the peer maps it +# silently corrupts the hop. +_PRODUCER_TENSORS: OrderedDict[tuple[bytes, int], torch.Tensor] = OrderedDict() +_WARNED_ASYNC_ALLOC = False + + +def _producer_key(handle: bytes, storage_offset_bytes: int) -> tuple[bytes, int]: + return (handle, storage_offset_bytes) + + +def _retain_producer_tensor( + handle: bytes, storage_offset_bytes: int, tensor: torch.Tensor +) -> None: + key = _producer_key(handle, storage_offset_bytes) + _PRODUCER_TENSORS[key] = tensor + _PRODUCER_TENSORS.move_to_end(key) + + +def release_retained_producer_tensors() -> None: + """Drop producer-side IPC retains after the peer has mapped them.""" + _PRODUCER_TENSORS.clear() + + +@dataclass +class CudaIpcRef: + """Picklable CUDA IPC handle plus enough metadata to rebuild the tensor.""" + + dtype: str + shape: tuple[int, ...] + stride: tuple[int, ...] + storage_offset: int + device_index: int + handle: bytes + storage_size_bytes: int + storage_offset_bytes: int + ref_counter_handle: bytes + ref_counter_offset: int + event_handle: bytes + event_sync_required: bool + + @classmethod + def from_tensor(cls, tensor: torch.Tensor) -> CudaIpcRef: + if not tensor.is_cuda: + raise TypeError("CudaIpcRef only shares CUDA tensors") + tensor = tensor.detach().contiguous() + ( + device_index, + handle, + storage_size_bytes, + storage_offset_bytes, + ref_counter_handle, + ref_counter_offset, + event_handle, + event_sync_required, + ) = tensor.untyped_storage()._share_cuda_() + ref = cls( + dtype=str(tensor.dtype).removeprefix("torch."), + shape=tuple(tensor.shape), + stride=tuple(tensor.stride()), + storage_offset=int(tensor.storage_offset()), + device_index=int(device_index), + handle=handle, + storage_size_bytes=int(storage_size_bytes), + storage_offset_bytes=int(storage_offset_bytes), + ref_counter_handle=ref_counter_handle, + ref_counter_offset=int(ref_counter_offset), + event_handle=event_handle, + event_sync_required=bool(event_sync_required), + ) + _retain_producer_tensor(handle, int(storage_offset_bytes), tensor) + return ref + + def materialize(self) -> torch.Tensor: + dtype = getattr(torch, self.dtype) + if self.handle is None or self.storage_size_bytes == 0: + return torch.empty( + self.shape, dtype=dtype, device=f"cuda:{self.device_index}" + ) + + local = _PRODUCER_TENSORS.pop( + _producer_key(self.handle, self.storage_offset_bytes), None + ) + if local is not None: + return local.detach().clone() + + torch.cuda._lazy_init() + storage = torch.UntypedStorage._new_shared_cuda( + self.device_index, + self.handle, + self.storage_size_bytes, + self.storage_offset_bytes, + self.ref_counter_handle, + self.ref_counter_offset, + self.event_handle, + self.event_sync_required, + ) + typed = torch.storage.TypedStorage( + wrap_storage=storage, dtype=dtype, _internal=True + ) + mapped = torch._utils._rebuild_tensor( + typed, self.storage_offset, self.shape, self.stride + ) + # Own a private copy so the IPC mapping can be released immediately. + return mapped.clone() + + +def spill_cuda_tensors(value: Any, *, in_place: bool = False) -> Any: + """Replace CUDA tensors with ``CudaIpcRef`` handles. + + By default dataclasses are shallow-copied so the caller's tensors stay + intact (needed on the client, which still holds the original ``Req``). + Scheduler replies can pass ``in_place=True``. + """ + return _map_tree(value, _spill_one, copy_dataclasses=not in_place) + + +def materialize_cuda_refs(value: Any) -> Any: + """Rebuild CUDA tensors from ``CudaIpcRef`` handles, in place on dataclasses.""" + return _map_tree(value, _materialize_one, copy_dataclasses=False) + + +def _spill_one(value: Any) -> Any: + if isinstance(value, torch.Tensor) and value.is_cuda: + try: + return CudaIpcRef.from_tensor(value) + except RuntimeError as exc: + # cudaMallocAsync (ComfyUI default) cannot export IPC handles. + # Leave the tensor in place so pickle falls back to a host copy. + msg = str(exc) + if "shareIpcHandle" not in msg and "cudaMallocAsync" not in msg: + raise + global _WARNED_ASYNC_ALLOC + if not _WARNED_ASYNC_ALLOC: + _WARNED_ASYNC_ALLOC = True + logger.warning( + "CUDA IPC export failed (%s); falling back to a host pickle " + "copy. ComfyUI's default cudaMallocAsync pool cannot export " + "IPC handles.", + exc, + ) + return value + + +def _materialize_one(value: Any) -> Any: + if isinstance(value, CudaIpcRef): + return value.materialize() + return value + + +def _map_tree(value: Any, fn, copy_dataclasses: bool) -> Any: + replaced = fn(value) + if replaced is not value: + return replaced + if isinstance(value, list): + return [_map_tree(item, fn, copy_dataclasses) for item in value] + if isinstance(value, tuple): + return tuple(_map_tree(item, fn, copy_dataclasses) for item in value) + if isinstance(value, dict): + return { + key: _map_tree(item, fn, copy_dataclasses) for key, item in value.items() + } + if dataclasses.is_dataclass(value) and not isinstance(value, type): + if copy_dataclasses: + value = copy.copy(value) + for field in dataclasses.fields(value): + current = getattr(value, field.name, None) + updated = _map_tree(current, fn, copy_dataclasses) + if updated is not current: + setattr(value, field.name, updated) + return value + return value + + +_SEP = "\x1f" + + +def detach_cuda_tensors(value: Any) -> tuple[Any, dict[str, torch.Tensor]]: + """Copy the tree, replace CUDA tensors with ``None``, collect them by path.""" + tensors: dict[str, torch.Tensor] = {} + skeleton = _detach(value, "", tensors) + return skeleton, tensors + + +def attach_cuda_tensors( + value: Any, + tensors: dict[str, torch.Tensor], + device: torch.device | None = None, +) -> Any: + """Put detached CUDA tensors back onto the skeleton.""" + if device is not None: + moved: dict[str, torch.Tensor] = {} + for key, tensor in tensors.items(): + moved[key] = ( + tensor + if tensor.device == device + else tensor.to(device, non_blocking=True) + ) + tensors = moved + return _attach(value, "", tensors) + + +def _join(prefix: str, part: str) -> str: + return part if not prefix else f"{prefix}{_SEP}{part}" + + +def _detach(value: Any, prefix: str, tensors: dict[str, torch.Tensor]) -> Any: + if isinstance(value, torch.Tensor) and value.is_cuda: + tensors[prefix] = value + return None + if isinstance(value, list): + return [ + _detach(item, _join(prefix, str(i)), tensors) + for i, item in enumerate(value) + ] + if isinstance(value, tuple): + return tuple( + _detach(item, _join(prefix, str(i)), tensors) + for i, item in enumerate(value) + ) + if isinstance(value, dict): + return { + key: _detach(item, _join(prefix, str(key)), tensors) + for key, item in value.items() + } + if dataclasses.is_dataclass(value) and not isinstance(value, type): + value = copy.copy(value) + for field in dataclasses.fields(value): + current = getattr(value, field.name, None) + updated = _detach(current, _join(prefix, field.name), tensors) + if updated is not current: + setattr(value, field.name, updated) + return value + return value + + +def _attach(value: Any, prefix: str, tensors: dict[str, torch.Tensor]) -> Any: + if prefix in tensors: + return tensors[prefix] + if isinstance(value, list): + return [ + _attach(item, _join(prefix, str(i)), tensors) + for i, item in enumerate(value) + ] + if isinstance(value, tuple): + return tuple( + _attach(item, _join(prefix, str(i)), tensors) + for i, item in enumerate(value) + ) + if isinstance(value, dict): + return { + key: _attach(item, _join(prefix, str(key)), tensors) + for key, item in value.items() + } + if dataclasses.is_dataclass(value) and not isinstance(value, type): + for field in dataclasses.fields(value): + current = getattr(value, field.name, None) + updated = _attach(current, _join(prefix, field.name), tensors) + if updated is not current: + setattr(value, field.name, updated) + return value + return value diff --git a/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/__init__.py b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/__init__.py new file mode 100644 index 0000000000..779af76a63 --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/__init__.py @@ -0,0 +1,34 @@ +# SPDX-License-Identifier: Apache-2.0 +"""ComfyUI single-file DiT checkpoints. + +``spec`` holds the shared checkpoint description and load path. Each sibling +module registers one DiT family. Importing this package registers every spec. +""" + +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( # noqa: F401 + flux, + minimax_h3, + qwen_image, + zimage, +) +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import ( + ComfyUICheckpointSpec, + ParamNamesMapping, + WeightIterator, + get_comfyui_checkpoint_spec, + get_registered_comfyui_pipeline_names, + is_comfyui_single_file, + load_comfyui_transformer, + register_comfyui_checkpoint, +) + +__all__ = [ + "ComfyUICheckpointSpec", + "ParamNamesMapping", + "WeightIterator", + "get_comfyui_checkpoint_spec", + "get_registered_comfyui_pipeline_names", + "is_comfyui_single_file", + "load_comfyui_transformer", + "register_comfyui_checkpoint", +] diff --git a/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/flux.py b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/flux.py new file mode 100644 index 0000000000..c2f4e17a9c --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/flux.py @@ -0,0 +1,262 @@ +# SPDX-License-Identifier: Apache-2.0 +"""ComfyUI Flux checkpoint spec.""" + +import re +from typing import Any + +import torch + +from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import ( + ComfyUICheckpointSpec, + WeightIterator, + register_comfyui_checkpoint, +) +from sglang.multimodal_gen.runtime.server_args import ServerArgs +from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger + +logger = init_logger(__name__) + + +def _build_dit_config(server_args: ServerArgs) -> FluxConfig: + dit_config = getattr(server_args.pipeline_config, "dit_config", None) + if not isinstance(dit_config, FluxConfig): + dit_config = FluxConfig() + server_args.pipeline_config.dit_config = dit_config + + # ComfyUI Flux checkpoints always carry guidance_in weights. + dit_config.arch_config.guidance_embeds = True + return dit_config + + +def _split_sizes(dit_config: FluxConfig) -> tuple[int, int]: + arch_config = dit_config.arch_config + hidden_size = arch_config.num_attention_heads * arch_config.attention_head_dim + mlp_hidden_dim = int(hidden_size * getattr(arch_config, "mlp_ratio", 4.0)) + return 3 * hidden_size, mlp_hidden_dim + + +def _split_qkv( + tensor: torch.Tensor, hidden_size: int +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + return ( + tensor[:hidden_size], + tensor[hidden_size : 2 * hidden_size], + tensor[2 * hidden_size : 3 * hidden_size], + ) + + +def _swap_halves(tensor: torch.Tensor) -> torch.Tensor: + """ComfyUI emits [shift, scale]; AdaLayerNormContinuous expects [scale, shift].""" + half = tensor.shape[0] // 2 + return torch.cat([tensor[half:], tensor[:half]], dim=0) + + +def _convert_weights(weights: WeightIterator, dit_config: Any) -> WeightIterator: + qkv_size, mlp_hidden_dim = _split_sizes(dit_config) + has_guidance_embeds = dit_config.arch_config.guidance_embeds + hidden_size = qkv_size // 3 + + for name, tensor in weights: + if not has_guidance_embeds and name.startswith("guidance_in."): + continue + + match = re.match( + r"double_blocks\.(\d+)\.(img_attn|txt_attn)\.qkv\.(weight|bias)$", name + ) + if match: + block_idx, attn_type, param_type = match.groups() + if tensor.shape[0] < qkv_size: + logger.warning( + "%s shape %s smaller than expected qkv size %d, skipping", + name, + tensor.shape, + qkv_size, + ) + continue + + q, k, v = _split_qkv(tensor, hidden_size) + prefix = f"transformer_blocks.{block_idx}.attn" + if attn_type == "img_attn": + yield f"{prefix}.to_q.{param_type}", q + yield f"{prefix}.to_k.{param_type}", k + yield f"{prefix}.to_v.{param_type}", v + else: + yield f"{prefix}.add_q_proj.{param_type}", q + yield f"{prefix}.add_k_proj.{param_type}", k + yield f"{prefix}.add_v_proj.{param_type}", v + continue + + match = re.match(r"single_blocks\.(\d+)\.linear1\.(weight|bias)$", name) + if match: + block_idx, param_type = match.groups() + expected_size = qkv_size + mlp_hidden_dim + if tensor.shape[0] < expected_size: + logger.warning( + "linear1.%s shape %s doesn't match expected size %d, skipping", + param_type, + tensor.shape, + expected_size, + ) + continue + + q, k, v = _split_qkv(tensor[:qkv_size], hidden_size) + prefix = f"single_transformer_blocks.{block_idx}" + yield f"{prefix}.attn.to_q.{param_type}", q + yield f"{prefix}.attn.to_k.{param_type}", k + yield f"{prefix}.attn.to_v.{param_type}", v + yield f"{prefix}.proj_mlp.{param_type}", tensor[qkv_size:] + continue + + if name in ( + "final_layer.adaLN_modulation.1.weight", + "final_layer.adaLN_modulation.1.bias", + ): + yield name, _swap_halves(tensor) + continue + + yield name, tensor + + +# ComfyUI names differ from SGLang's diffusers-style names. Fused tensors that +# _convert_weights already split are emitted under their SGLang names, so only +# the untouched ones need an entry here. +_PARAM_NAMES_MAPPING = { + r"double_blocks\.(\d+)\.img_attn\.proj\.(weight|bias)$": ( + r"transformer_blocks.\1.attn.to_out.0.\2", + None, + None, + ), + r"double_blocks\.(\d+)\.txt_attn\.proj\.(weight|bias)$": ( + r"transformer_blocks.\1.attn.to_add_out.\2", + None, + None, + ), + r"double_blocks\.(\d+)\.img_attn\.norm\.query_norm\.scale$": ( + r"transformer_blocks.\1.attn.norm_q.weight", + None, + None, + ), + r"double_blocks\.(\d+)\.img_attn\.norm\.key_norm\.scale$": ( + r"transformer_blocks.\1.attn.norm_k.weight", + None, + None, + ), + r"double_blocks\.(\d+)\.txt_attn\.norm\.query_norm\.scale$": ( + r"transformer_blocks.\1.attn.norm_added_q.weight", + None, + None, + ), + r"double_blocks\.(\d+)\.txt_attn\.norm\.key_norm\.scale$": ( + r"transformer_blocks.\1.attn.norm_added_k.weight", + None, + None, + ), + r"double_blocks\.(\d+)\.img_mlp\.0\.(weight|bias)$": ( + r"transformer_blocks.\1.ff.net.0.proj.\2", + None, + None, + ), + r"double_blocks\.(\d+)\.img_mlp\.2\.(weight|bias)$": ( + r"transformer_blocks.\1.ff.net.2.\2", + None, + None, + ), + r"double_blocks\.(\d+)\.txt_mlp\.0\.(weight|bias)$": ( + r"transformer_blocks.\1.ff_context.net.0.proj.\2", + None, + None, + ), + r"double_blocks\.(\d+)\.txt_mlp\.2\.(weight|bias)$": ( + r"transformer_blocks.\1.ff_context.net.2.\2", + None, + None, + ), + r"double_blocks\.(\d+)\.img_mod\.lin\.(weight|bias)$": ( + r"transformer_blocks.\1.norm1.linear.\2", + None, + None, + ), + r"double_blocks\.(\d+)\.txt_mod\.lin\.(weight|bias)$": ( + r"transformer_blocks.\1.norm1_context.linear.\2", + None, + None, + ), + r"single_blocks\.(\d+)\.linear2\.(weight|bias)$": ( + r"single_transformer_blocks.\1.proj_out.\2", + None, + None, + ), + r"single_blocks\.(\d+)\.norm\.query_norm\.scale$": ( + r"single_transformer_blocks.\1.attn.norm_q.weight", + None, + None, + ), + r"single_blocks\.(\d+)\.norm\.key_norm\.scale$": ( + r"single_transformer_blocks.\1.attn.norm_k.weight", + None, + None, + ), + r"single_blocks\.(\d+)\.modulation\.lin\.(weight|bias)$": ( + r"single_transformer_blocks.\1.norm.linear.\2", + None, + None, + ), + r"^time_in\.in_layer\.(weight|bias)$": ( + r"time_text_embed.timestep_embedder.linear_1.\1", + None, + None, + ), + r"^time_in\.out_layer\.(weight|bias)$": ( + r"time_text_embed.timestep_embedder.linear_2.\1", + None, + None, + ), + r"^txt_in\.(weight|bias)$": (r"context_embedder.\1", None, None), + r"^vector_in\.in_layer\.(weight|bias)$": ( + r"time_text_embed.text_embedder.linear_1.\1", + None, + None, + ), + r"^vector_in\.out_layer\.(weight|bias)$": ( + r"time_text_embed.text_embedder.linear_2.\1", + None, + None, + ), + r"^final_layer\.linear\.(weight|bias)$": (r"proj_out.\1", None, None), + r"^final_layer\.norm_final\.(weight|bias)$": (r"norm_out.\1", None, None), + r"^final_layer\.adaLN_modulation\.1\.(weight|bias)$": ( + r"norm_out.linear.\1", + None, + None, + ), + r"^img_in\.(weight|bias)$": (r"x_embedder.\1", None, None), + r"^guidance_in\.in_layer\.(weight|bias)$": ( + r"time_text_embed.guidance_embedder.linear_1.\1", + None, + None, + ), + r"^guidance_in\.out_layer\.(weight|bias)$": ( + r"time_text_embed.guidance_embedder.linear_2.\1", + None, + None, + ), +} + + +register_comfyui_checkpoint( + "FluxPipeline", + ComfyUICheckpointSpec( + dit_cls_name="FluxTransformer2DModel", + build_dit_config=_build_dit_config, + param_names_mapping=_PARAM_NAMES_MAPPING, + convert_weights=_convert_weights, + # ComfyUI Flux checkpoints ship without the optional attention biases. + strict=False, + # FluxConfig's own mapping reads the same BFL names this spec does, but + # targets a different parameter layout (fused to_qkv, ff.linear_in, + # time_guidance_embed) than FluxTransformer2DModel exposes. Layering the + # two rewrites correctly resolved names into ones the model lacks. + inherit_config_mapping=False, + ), +) diff --git a/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/minimax_h3.py b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/minimax_h3.py new file mode 100644 index 0000000000..258f07f285 --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/minimax_h3.py @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: Apache-2.0 +"""ComfyUI MiniMax-H3 checkpoint spec. + +The Comfy-Org BF16 FL2VA file uses ComfyUI parameter names and already stores +fused QKV as ``[q_all, k_all, v_all]``. Identity name mapping is enough, but +the DiT's default HF weight loader must not run the grouped→concat reorder. +""" + +from sglang.multimodal_gen.configs.models.dits.minimax_h3 import MiniMaxH3DiTConfig +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import ( + ComfyUICheckpointSpec, + register_comfyui_checkpoint, +) +from sglang.multimodal_gen.runtime.server_args import ServerArgs + + +def _build_dit_config(server_args: ServerArgs) -> MiniMaxH3DiTConfig: + dit_config = getattr(server_args.pipeline_config, "dit_config", None) + if not isinstance(dit_config, MiniMaxH3DiTConfig): + dit_config = MiniMaxH3DiTConfig() + server_args.pipeline_config.dit_config = dit_config + # ComfyUI / Comfy-Org files match the runtime QKV layout. Native + # ``sglang serve`` keeps the default grouped HF reorder. + dit_config.arch_config.qkv_checkpoint_grouped = False + return dit_config + + +register_comfyui_checkpoint( + "MiniMaxH3Pipeline", + ComfyUICheckpointSpec( + dit_cls_name="MiniMaxH3DiTModel", + build_dit_config=_build_dit_config, + inherit_config_mapping=True, + ), +) diff --git a/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/qwen_image.py b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/qwen_image.py new file mode 100644 index 0000000000..769ed8d349 --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/qwen_image.py @@ -0,0 +1,57 @@ +# SPDX-License-Identifier: Apache-2.0 +"""ComfyUI Qwen-Image checkpoint specs (text-to-image and edit).""" + +from collections.abc import Callable + +from sglang.multimodal_gen.configs.models.dits.qwenimage import ( + QwenImageArchConfig, + QwenImageDitConfig, +) +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import ( + ComfyUICheckpointSpec, + register_comfyui_checkpoint, +) +from sglang.multimodal_gen.runtime.server_args import ServerArgs + + +def _dit_config_builder( + zero_cond_t: bool, +) -> Callable[[ServerArgs], QwenImageDitConfig]: + def build(server_args: ServerArgs) -> QwenImageDitConfig: + # ComfyUI checkpoints carry no config, so the architecture is pinned here. + dit_config = QwenImageDitConfig( + arch_config=QwenImageArchConfig( + patch_size=2, + in_channels=64, + out_channels=16, + num_layers=60, + attention_head_dim=128, + num_attention_heads=24, + joint_attention_dim=3584, + pooled_projection_dim=768, + guidance_embeds=False, + axes_dims_rope=(16, 56, 56), + zero_cond_t=zero_cond_t, + ) + ) + server_args.pipeline_config.dit_config = dit_config + return dit_config + + return build + + +_PARAM_NAMES_MAPPING = {r"^model\.diffusion_model\.(.*)$": (r"\1", None, None)} + + +for _pipeline_name, _zero_cond_t in ( + ("QwenImagePipeline", False), + ("QwenImageEditPlusPipeline", True), +): + register_comfyui_checkpoint( + _pipeline_name, + ComfyUICheckpointSpec( + dit_cls_name="QwenImageTransformer2DModel", + build_dit_config=_dit_config_builder(_zero_cond_t), + param_names_mapping=_PARAM_NAMES_MAPPING, + ), + ) diff --git a/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/spec.py b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/spec.py new file mode 100644 index 0000000000..34aae2426a --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/spec.py @@ -0,0 +1,304 @@ +# SPDX-License-Identifier: Apache-2.0 +"""ComfyUI single-file DiT checkpoints: spec, registry, and load. + +ComfyUI ships a DiT as one ``.safetensors`` (or a GGUF override) with no +``model_index.json`` and its own parameter names. A spec supplies what the +shared loader cannot infer: which DiT config to build, how names map onto +SGLang, and how to reshape tensors whose layout differs. Per-model specs live +in the sibling modules of this package. + +Everything else -- meta-device init, FSDP sharding, quantization, CPU +offload -- goes through the regular transformer load path. +""" + +from __future__ import annotations + +import os +from collections.abc import Callable, Iterator +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any + +import torch + +from sglang.multimodal_gen.runtime.distributed import get_local_torch_device +from sglang.multimodal_gen.runtime.loader.fsdp_load import maybe_load_fsdp_model +from sglang.multimodal_gen.runtime.loader.weight_utils import ( + safetensors_weights_iterator, +) +from sglang.multimodal_gen.runtime.models.registry import ModelRegistry +from sglang.multimodal_gen.runtime.server_args import ServerArgs +from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger +from sglang.multimodal_gen.runtime.utils.precision import resolve_precision + +if TYPE_CHECKING: + from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import ( + ComposedPipelineBase, + ) + +logger = init_logger(__name__) + +WeightIterator = Iterator[tuple[str, torch.Tensor]] + +# ComfyUI mapping entries reuse the param_names_mapping format: +# source_regex -> (target_template, merge_index, num_params_to_merge) +ParamNamesMapping = dict[str, tuple[str, int | None, int | None]] + + +@dataclass(frozen=True) +class ComfyUICheckpointSpec: + """Per-model knowledge needed to load a ComfyUI checkpoint.""" + + dit_cls_name: str + build_dit_config: Callable[[ServerArgs], Any] + param_names_mapping: ParamNamesMapping = field(default_factory=dict) + # Reshapes tensors that param_names_mapping cannot express. Receives the raw + # safetensors iterator plus the built dit config, yields SGLang-shaped pairs. + convert_weights: Callable[[WeightIterator, Any], WeightIterator] | None = None + # Set False for checkpoints that legitimately omit parameters the model + # declares, such as optional biases. + strict: bool = True + # Whether to layer param_names_mapping on top of the DiT config's own + # mapping. Keep it True when the two act on different names (the config + # rules then finish the job, e.g. merging split QKV back into a fused + # parameter). Set False when both claim the same source names, since name + # mapping is applied repeatedly until it reaches a fixed point and the + # config rules would rewrite names this spec already resolved. + inherit_config_mapping: bool = True + + +_SPEC_REGISTRY: dict[str, ComfyUICheckpointSpec] = {} +_SPECS_DISCOVERED = False + + +def register_comfyui_checkpoint( + pipeline_name: str, spec: ComfyUICheckpointSpec +) -> None: + _SPEC_REGISTRY[pipeline_name] = spec + + +def _discover_checkpoint_specs() -> None: + global _SPECS_DISCOVERED + if _SPECS_DISCOVERED: + return + _SPECS_DISCOVERED = True + from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( # noqa: F401 + flux, + minimax_h3, + qwen_image, + zimage, + ) + + +def get_comfyui_checkpoint_spec(pipeline_name: str) -> ComfyUICheckpointSpec | None: + _discover_checkpoint_specs() + return _SPEC_REGISTRY.get(pipeline_name) + + +def get_registered_comfyui_pipeline_names() -> list[str]: + _discover_checkpoint_specs() + return sorted(_SPEC_REGISTRY) + + +def is_comfyui_single_file(model_path: str) -> bool: + """ComfyUI ships DiTs as one safetensors file with no model_index.json.""" + return os.path.isfile(model_path) and model_path.endswith(".safetensors") + + +def _load_comfyui_gguf_transformer( + *, + model_cls, + dit_config, + gguf_file: str, + server_args: ServerArgs, + mapping: dict[str, Any], + spec, +): + from sglang.multimodal_gen.runtime.layers.quantization.gguf import GGUFConfig + from sglang.multimodal_gen.runtime.loader.gguf_weights import ( + gguf_weights_iterator, + read_gguf_tensor_meta, + ) + from sglang.multimodal_gen.runtime.loader.minimax_h3_weights import ( + validate_minimax_h3_checkpoint_variant, + ) + from sglang.multimodal_gen.runtime.loader.transformer_load_utils import ( + _validate_gguf_runtime_support, + ) + + _validate_gguf_runtime_support(server_args, "transformer") + if spec.dit_cls_name == "MiniMaxH3DiTModel": + validate_minimax_h3_checkpoint_variant( + [gguf_file], str(server_args.model_variant or "fl2va") + ) + + tensor_meta = read_gguf_tensor_meta(gguf_file) + curve = tensor_meta.get("adaln_t_table") + if curve is not None: + if curve.is_quantized or len(curve.logical_shape) != 2: + raise ValueError( + "MiniMax-H3 adaln_t_table must be an unquantized 2D tensor" + ) + curve_grid, time_embed_dim = curve.logical_shape + if curve_grid < 2: + raise ValueError("MiniMax-H3 adaln_t_table needs at least two rows") + dit_config.arch_config.adaln_curve_grid = curve_grid + dit_config.arch_config.time_embed_dim = time_embed_dim + + quant_config = GGUFConfig(gguf_file=gguf_file, tensor_meta=tensor_meta) + packed = getattr(model_cls, "packed_modules_mapping", None) + if packed: + quant_config.packed_modules_mapping = packed + + logger.info("Loading %s from GGUF file %s", spec.dit_cls_name, gguf_file) + original_mapping = model_cls.param_names_mapping + model_cls.param_names_mapping = mapping + try: + return maybe_load_fsdp_model( + model_cls=model_cls, + init_params={ + "config": dit_config, + "hf_config": {}, + "quant_config": quant_config, + }, + weight_dir_list=[], + device=get_local_torch_device(), + hsdp_replicate_dim=server_args.hsdp_replicate_dim, + hsdp_shard_dim=server_args.hsdp_shard_dim, + component_starts_on_cpu=server_args.should_start_component_on_cpu( + "transformer" + ), + pin_cpu_memory=server_args.pin_cpu_memory, + fsdp_inference=server_args.should_use_fsdp_for_component("transformer"), + param_dtype=None, + reduce_dtype=torch.float32, + output_dtype=None, + strict=False, + weights_iterator=gguf_weights_iterator(gguf_file, tensor_meta), + ) + finally: + model_cls.param_names_mapping = original_mapping + + +def load_comfyui_transformer( + pipeline: ComposedPipelineBase, + server_args: ServerArgs, + loaded_modules: dict[str, torch.nn.Module] | None = None, +) -> dict[str, Any]: + """Load the DiT from a single ComfyUI safetensors file. + + Reuses the shared FSDP-aware transformer load path; the spec only supplies + what the file itself cannot describe. A GGUF + ``--transformer-weights-path`` replaces only the DiT weights. + """ + if loaded_modules is not None and "transformer" in loaded_modules: + return { + "transformer": loaded_modules["transformer"], + "scheduler": pipeline.get_module("scheduler"), + } + + spec = get_comfyui_checkpoint_spec(pipeline.pipeline_name) + if spec is None: + raise ValueError( + f"{pipeline.pipeline_name} has no ComfyUI checkpoint spec, so it cannot " + f"load a single safetensors file. Pipelines with a spec: " + f"{get_registered_comfyui_pipeline_names()}" + ) + + model_path = pipeline.model_path + dit_config = spec.build_dit_config(server_args) + mapping = dict(spec.param_names_mapping) + if spec.inherit_config_mapping: + mapping = { + **(dit_config.arch_config.param_names_mapping or {}), + **mapping, + } + dit_config.arch_config.param_names_mapping = mapping + + model_cls, _ = ModelRegistry.resolve_model_cls(spec.dit_cls_name) + param_dtype = resolve_precision(server_args, "dit", precision_attr="dit_precision") + server_args.model_paths["transformer"] = os.path.dirname(model_path) or "." + + from sglang.multimodal_gen.runtime.loader.transformer_load_utils import ( + resolve_transformer_gguf_to_load, + ) + + gguf_file = resolve_transformer_gguf_to_load(server_args, "transformer") + if gguf_file is not None: + model = _load_comfyui_gguf_transformer( + model_cls=model_cls, + dit_config=dit_config, + gguf_file=gguf_file, + server_args=server_args, + mapping=mapping, + spec=spec, + ) + else: + # Only override the iterator when tensors need reshaping; leaving it None + # keeps the rank-local checkpoint fast path available. + weights_iterator = None + if spec.convert_weights is not None: + weights_iterator = spec.convert_weights( + safetensors_weights_iterator([model_path]), dit_config + ) + + # GGUF already sets AdaLN curve from tensor meta. Pruned BF16 + # safetensors keep the same adaln_t_table; without this the DiT is + # built as the unpruned MLP and load fails on that extra parameter. + if spec.dit_cls_name == "MiniMaxH3DiTModel": + from sglang.multimodal_gen.runtime.loader.minimax_h3_weights import ( + inspect_minimax_h3_safetensors, + ) + + adaln_curve_shape, _ = inspect_minimax_h3_safetensors([model_path]) + if adaln_curve_shape is not None: + ( + dit_config.arch_config.adaln_curve_grid, + dit_config.arch_config.time_embed_dim, + ) = adaln_curve_shape + logger.info( + "MiniMax-H3 ComfyUI checkpoint uses AdaLN curve %s", + adaln_curve_shape, + ) + + logger.info( + "Loading %s from ComfyUI checkpoint %s, param_dtype: %s", + spec.dit_cls_name, + model_path, + param_dtype, + ) + + # Weight loading reads param_names_mapping off the model, which inherits it + # from the class, so the ComfyUI names have to be visible for the whole load. + original_mapping = model_cls.param_names_mapping + model_cls.param_names_mapping = mapping + try: + model = maybe_load_fsdp_model( + model_cls=model_cls, + init_params={"config": dit_config, "hf_config": {}}, + weight_dir_list=[model_path], + device=get_local_torch_device(), + hsdp_replicate_dim=server_args.hsdp_replicate_dim, + hsdp_shard_dim=server_args.hsdp_shard_dim, + component_starts_on_cpu=server_args.should_start_component_on_cpu( + "transformer" + ), + pin_cpu_memory=server_args.pin_cpu_memory, + fsdp_inference=server_args.should_use_fsdp_for_component("transformer"), + param_dtype=param_dtype, + reduce_dtype=torch.float32, + output_dtype=None, + strict=spec.strict, + weights_iterator=weights_iterator, + ) + finally: + model_cls.param_names_mapping = original_mapping + + for param in model.parameters(): + param.requires_grad = False + + logger.info( + "Loaded transformer with %.2fB parameters", + sum(p.numel() for p in model.parameters()) / 1e9, + ) + + return {"transformer": model, "scheduler": pipeline.get_module("scheduler")} diff --git a/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/zimage.py b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/zimage.py new file mode 100644 index 0000000000..6347ddbf67 --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/loader/comfyui_checkpoints/zimage.py @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: Apache-2.0 +"""ComfyUI Z-Image (lumina2) checkpoint spec.""" + +import re +from typing import Any + +from sglang.multimodal_gen.configs.models.dits.zimage import ZImageDitConfig +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import ( + ComfyUICheckpointSpec, + WeightIterator, + register_comfyui_checkpoint, +) +from sglang.multimodal_gen.runtime.server_args import ServerArgs + + +def _build_dit_config(server_args: ServerArgs) -> ZImageDitConfig: + dit_config = getattr(server_args.pipeline_config, "dit_config", None) + if not isinstance(dit_config, ZImageDitConfig): + dit_config = ZImageDitConfig() + server_args.pipeline_config.dit_config = dit_config + return dit_config + + +def _convert_weights(weights: WeightIterator, dit_config: Any) -> WeightIterator: + """Split ComfyUI's merged attention qkv into to_q / to_k / to_v.""" + arch_config = dit_config.arch_config + q_size = arch_config.dim + k_size = ( + arch_config.dim // arch_config.num_attention_heads + ) * arch_config.n_kv_heads + + for name, tensor in weights: + match = re.match( + r"(layers|noise_refiner|context_refiner)\.(\d+)\.attention\.qkv\.(weight|bias)$", + name, + ) + if not match: + yield name, tensor + continue + + module_name, layer_idx, param_type = match.groups() + prefix = f"{module_name}.{layer_idx}.attention" + yield f"{prefix}.to_q.{param_type}", tensor[:q_size] + yield f"{prefix}.to_k.{param_type}", tensor[q_size : q_size + k_size] + yield f"{prefix}.to_v.{param_type}", tensor[q_size + k_size :] + + +_PARAM_NAMES_MAPPING = { + r"(.*)\.attention\.k_norm\.weight$": (r"\1.attention.norm_k.weight", None, None), + r"(.*)\.attention\.q_norm\.weight$": (r"\1.attention.norm_q.weight", None, None), + r"(.*)\.attention\.out\.weight$": (r"\1.attention.to_out.0.weight", None, None), + r"^final_layer\.(.*)$": (r"all_final_layer.2-1.\1", None, None), + r"^x_embedder\.(.*)$": (r"all_x_embedder.2-1.\1", None, None), +} + + +register_comfyui_checkpoint( + "ZImagePipeline", + ComfyUICheckpointSpec( + dit_cls_name="ZImageTransformer2DModel", + build_dit_config=_build_dit_config, + param_names_mapping=_PARAM_NAMES_MAPPING, + convert_weights=_convert_weights, + ), +) diff --git a/python/sglang/multimodal_gen/runtime/loader/rank_local_checkpoint.py b/python/sglang/multimodal_gen/runtime/loader/rank_local_checkpoint.py index 47a1ee12a2..a5334dc9f8 100644 --- a/python/sglang/multimodal_gen/runtime/loader/rank_local_checkpoint.py +++ b/python/sglang/multimodal_gen/runtime/loader/rank_local_checkpoint.py @@ -15,6 +15,7 @@ from torch.distributed.fsdp import FSDPModule from sglang.multimodal_gen.runtime.distributed import get_tp_rank, get_tp_world_size from sglang.multimodal_gen.runtime.layers.linear import ( ColumnParallelLinear, + MergedColumnParallelLinear, ReplicatedLinear, RowParallelLinear, ) @@ -236,6 +237,17 @@ def read_rank_local_tensor( return local_tensor +def _merged_output_sizes(actual_param: torch.nn.Parameter) -> list[int] | None: + """Logical output widths of a fused-on-disk MergedColumnParallelLinear.""" + weight_loader = actual_param.__dict__.get("weight_loader") + if not isinstance(weight_loader, MethodType): + return None + owner = weight_loader.__self__ + if isinstance(owner, MergedColumnParallelLinear): + return list(owner.output_sizes) + return None + + def _resolve_tp_shard_dim( actual_param: torch.nn.Parameter, ) -> tuple[bool, int | None]: @@ -318,6 +330,7 @@ def read_tp_local_tensor( shard_dim: int | None, tp_rank: int, tp_size: int, + output_sizes: list[int] | None = None, ) -> torch.Tensor: if shard_dim is None: assembled_shape = assembled_source_shape(sources) @@ -337,6 +350,37 @@ def read_tp_local_tensor( source.merge_index if source.merge_index is not None else 0, ), ) + # Fused-on-disk [Q|K|V] / [gate|up]: each logical matrix is sharded on + # its own, then concatenated. A contiguous half of the fused tensor is + # the same shape as the TP-local parameter but the wrong values. + if ( + output_sizes is not None + and len(ordered_sources) == 1 + and all(size % tp_size == 0 for size in output_sizes) + ): + source = ordered_sources[0] + if source.shape[shard_dim] != sum(output_sizes): + raise RuntimeError( + "Merged TP shard sizes do not cover the fused checkpoint: " + f"output_sizes={output_sizes}, source_dim={source.shape[shard_dim]}" + ) + local_parts = [] + offset = 0 + for output_size in output_sizes: + shard_size = output_size // tp_size + slices = [slice(None)] * len(source.shape) + slices[shard_dim] = slice( + offset + tp_rank * shard_size, + offset + tp_rank * shard_size + shard_size, + ) + local_parts.append( + handles[source.file_path] + .get_slice(source.param_name)[tuple(slices)] + .contiguous() + ) + offset += output_size + return torch.cat(local_parts, dim=shard_dim) + local_parts = [] for source in ordered_sources: shard_size = source.shape[shard_dim] // tp_size @@ -380,6 +424,7 @@ def try_load_rank_local_tp_state_dict( sources_by_target, reverse_param_names_mapping = checkpoint_sources shard_dims: dict[str, int | None] = {} + output_sizes_by_name: dict[str, list[int] | None] = {} for target_param_name, sources in sources_by_target.items(): meta_param = meta_sd.get(target_param_name) if meta_param is None or isinstance(meta_param, dist_tensor.DTensor): @@ -394,15 +439,22 @@ def try_load_rank_local_tp_state_dict( param_dict, target_param_name, ) + output_sizes = None if actual_param is None: supported, shard_dim = True, None else: supported, shard_dim = _resolve_tp_shard_dim(actual_param) + output_sizes = _merged_output_sizes(actual_param) + if output_sizes is not None and any( + size % tp_size for size in output_sizes + ): + return None if not supported: return None if tp_local_shape(sources, shard_dim, tp_size) != tuple(meta_param.shape): return None shard_dims[target_param_name] = shard_dim + output_sizes_by_name[target_param_name] = output_sizes local_param_sd: dict[str, LocalTPShard] = {} local_bytes = 0 @@ -421,6 +473,7 @@ def try_load_rank_local_tp_state_dict( shard_dims[target_param_name], tp_rank, tp_size, + output_sizes=output_sizes_by_name[target_param_name], ) local_param_sd[target_param_name] = LocalTPShard(tensor) local_bytes += tensor.numel() * tensor.element_size() diff --git a/python/sglang/multimodal_gen/runtime/managers/scheduler.py b/python/sglang/multimodal_gen/runtime/managers/scheduler.py index bfca31ac39..f2eb77b07c 100644 --- a/python/sglang/multimodal_gen/runtime/managers/scheduler.py +++ b/python/sglang/multimodal_gen/runtime/managers/scheduler.py @@ -16,6 +16,11 @@ from sglang.multimodal_gen.runtime.disaggregation.roles import RoleType from sglang.multimodal_gen.runtime.disaggregation.scheduler_mixin import ( SchedulerDisaggMixin, ) +from sglang.multimodal_gen.runtime.distributed.ipc_cuda import ( + materialize_cuda_refs, + release_retained_producer_tensors, + spill_cuda_tensors, +) from sglang.multimodal_gen.runtime.distributed.utils import broadcast_pyobj from sglang.multimodal_gen.runtime.entrypoints.control_requests import ( GetDisaggStatsReq, @@ -771,6 +776,13 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag output_batch.output ) + with self._record_return_stage( + output_batch, "Scheduler.return_result.spill_cuda" + ): + # The previous reply has already been mapped: the client + # only sends the next hop after materializing the last one. + release_retained_producer_tensors() + spill_cuda_tensors(output_batch, in_place=True) with self._record_return_stage( output_batch, "Scheduler.return_result.pickle" ): @@ -1183,7 +1195,9 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag def recv_reqs(self) -> List[tuple[bytes, Any]]: """ - For non-main schedulers, reqs are broadcasted from main using broadcast_pyobj + For non-main schedulers, reqs are broadcasted from main using + broadcast_pyobj. ``--comfyui-mode`` multi-rank instead keeps CUDA + tensors on NCCL so per-step latents do not pickle onto the gloo group. """ if self.receiver is not None: try: @@ -1208,30 +1222,37 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag else: recv_reqs = None - # TODO: fix this condition - if self.server_args.sp_degree != 1: - recv_reqs = broadcast_pyobj( - recv_reqs, - self.worker.sp_group.rank, - self.worker.sp_cpu_group, - src=self.worker.sp_group.ranks[0], - ) + # Rebuild CUDA IPC handles on rank 0 (no-op when the payload has none). + if recv_reqs is not None: + recv_reqs = materialize_cuda_refs(recv_reqs) - if self.server_args.enable_cfg_parallel: - recv_reqs = broadcast_pyobj( - recv_reqs, - self.worker.cfg_group.rank, - self.worker.cfg_cpu_group, - src=self.worker.cfg_group.ranks[0], - ) + if self.server_args.comfyui_mode and self._is_multi_rank(): + recv_reqs = self._broadcast_recv_reqs(recv_reqs) + else: + # TODO: fix this condition + if self.server_args.sp_degree != 1: + recv_reqs = broadcast_pyobj( + recv_reqs, + self.worker.sp_group.rank, + self.worker.sp_cpu_group, + src=self.worker.sp_group.ranks[0], + ) - if self.server_args.tp_size > 1: - recv_reqs = broadcast_pyobj( - recv_reqs, - self.worker.tp_group.rank, - self.worker.tp_cpu_group, - src=self.worker.tp_group.ranks[0], - ) + if self.server_args.enable_cfg_parallel: + recv_reqs = broadcast_pyobj( + recv_reqs, + self.worker.cfg_group.rank, + self.worker.cfg_cpu_group, + src=self.worker.cfg_group.ranks[0], + ) + + if self.server_args.tp_size > 1: + recv_reqs = broadcast_pyobj( + recv_reqs, + self.worker.tp_group.rank, + self.worker.tp_cpu_group, + src=self.worker.tp_group.ranks[0], + ) assert recv_reqs is not None diff --git a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py b/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py index 29e8eaa01b..0d47d17f71 100644 --- a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py +++ b/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py @@ -822,8 +822,9 @@ class MiniMaxH3Attention(nn.Module): quant_config=quant_config, prefix=f"{prefix}.qkv_proj", ) - # Official safetensors interleave Q/K/V by head. Comfy and GGUF - # checkpoints already store [q_all, k_all, v_all]. + # Official safetensors interleave Q/K/V by head. Comfy BF16, Comfy + # convrot int8, and GGUF already store [q_all, k_all, v_all]. + skip_grouped_reorder = not getattr(arch, "qkv_checkpoint_grouped", True) checkpoint_qkv_is_native = quant_config is not None and ( quant_config.get_name() == "gguf" or quant_config.checkpoint_uses_native_qkv_layout @@ -831,7 +832,7 @@ class MiniMaxH3Attention(nn.Module): checkpoint_qkv_is_native = ( checkpoint_qkv_is_native or arch.checkpoint_uses_diffusers_layout ) - if not checkpoint_qkv_is_native: + if not skip_grouped_reorder and not checkpoint_qkv_is_native: self._install_qkv_weight_loader(arch) self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps) self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps) diff --git a/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_comfyui_passthrough.py b/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_comfyui_passthrough.py index e87f558b8b..a0ba9310fa 100644 --- a/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_comfyui_passthrough.py +++ b/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_comfyui_passthrough.py @@ -112,7 +112,9 @@ class ComfyUIPassThroughScheduler(BaseScheduler, ConfigMixin, SchedulerMixin): Returns: The input sample unchanged (prev_sample = sample) """ - # Increment step index for tracking + # DenoisingStage resets _step_index to None before each request. + if self._step_index is None: + self._step_index = 0 self._step_index += 1 # Simply return the input sample unchanged diff --git a/python/sglang/multimodal_gen/runtime/pipelines/comfyui_flux_pipeline.py b/python/sglang/multimodal_gen/runtime/pipelines/comfyui_flux_pipeline.py deleted file mode 100644 index db7305d6a7..0000000000 --- a/python/sglang/multimodal_gen/runtime/pipelines/comfyui_flux_pipeline.py +++ /dev/null @@ -1,701 +0,0 @@ -# Copied and adapted from: https://github.com/hao-ai-lab/FastVideo -# SPDX-License-Identifier: Apache-2.0 - -import os -import re -from typing import Any, Generator - -import torch - -from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig -from sglang.multimodal_gen.runtime.distributed import get_local_torch_device -from sglang.multimodal_gen.runtime.models.registry import ModelRegistry -from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import ( - ComfyUIPassThroughScheduler, -) -from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline -from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import ( - ComposedPipelineBase, -) -from sglang.multimodal_gen.runtime.pipelines_core.stages import ( - ComfyUILatentPreparationStage, - DenoisingStage, -) -from sglang.multimodal_gen.runtime.server_args import ServerArgs -from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger -from sglang.multimodal_gen.runtime.utils.precision import resolve_precision - -logger = init_logger(__name__) - - -class ComfyUIFluxPipeline(LoRAPipeline, ComposedPipelineBase): - """ - Simplified pipeline for ComfyUI integration with only denoising stage. - - This pipeline requires pre-processed inputs: - - prompt_embeds: Pre-encoded text embeddings (list of tensors) - - negative_prompt_embeds: Pre-encoded negative prompt embeddings (if using CFG) - - latents: Optional initial noise latents (will be generated if not provided) - - Usage: - generator = DiffGenerator.from_pretrained( - model_path="path/to/model", - pipeline_class_name="ComfyUIFluxPipeline", - device="cuda", - ) - """ - - pipeline_name = "ComfyUIFluxPipeline" - - # Configuration classes for safetensors files without model_index.json - from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig - from sglang.multimodal_gen.configs.sample.flux import FluxSamplingParams - - pipeline_config_cls = FluxPipelineConfig - sampling_params_cls = FluxSamplingParams - - _required_config_modules = [ - "transformer", - "scheduler", - ] - - def initialize_pipeline(self, server_args: ServerArgs): - """ - Initialize the pipeline with ComfyUI pass-through scheduler. - This scheduler does not modify latents, allowing ComfyUI to handle denoising. - """ - self.modules["scheduler"] = ComfyUIPassThroughScheduler( - num_train_timesteps=1000 - ) - - if hasattr(server_args.pipeline_config, "vae_config"): - vae_config = server_args.pipeline_config.vae_config - if hasattr(vae_config, "post_init") and not hasattr( - vae_config, "_post_init_called" - ): - vae_config.post_init() - logger.info( - "Called vae_config.post_init() to set spatial_compression_ratio. " - f"spatial_compression_ratio={vae_config.arch_config.spatial_compression_ratio}" - ) - - def load_modules( - self, - server_args: ServerArgs, - loaded_modules: dict[str, torch.nn.Module] | None = None, - ) -> dict[str, Any]: - """ - Load modules for ComfyUIFluxPipeline. - - If model_path is a safetensors file, load transformer directly from it - without requiring model_index.json. Otherwise, fall back to default loading. - """ - if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"): - logger.info( - "Detected safetensors file, loading transformer directly from: %s", - self.model_path, - ) - return self._load_transformer_from_safetensors(server_args, loaded_modules) - else: - logger.info( - "Model path is a directory, using default loading method: %s", - self.model_path, - ) - return super().load_modules(server_args, loaded_modules) - - def _load_and_convert_weights_from_safetensors( - self, - model_cls: type, - dit_config: FluxConfig, - hf_config: dict, - safetensors_list: list[str], - updated_mapping: dict, - qkv_size: int, - mlp_hidden_dim: int, - has_guidance_embeds: bool, - default_dtype: torch.dtype, - ) -> tuple[torch.nn.Module, dict]: - """ - Load and convert weights from safetensors file, then load them into the model. - """ - from sglang.multimodal_gen.runtime.loader.utils import ( - get_param_names_mapping, - set_default_torch_dtype, - ) - from sglang.multimodal_gen.runtime.loader.weight_utils import ( - safetensors_weights_iterator, - ) - - logger.info( - "Converting ComfyUI Flux weights to SGLang format and loading model..." - ) - - # Create model on target device - device = get_local_torch_device() - with set_default_torch_dtype(default_dtype): - model = model_cls(**{"config": dit_config, "hf_config": hf_config}) - model = model.to(device) - - # Verify model has guidance_embedder if config says it should - has_guidance_embedder = hasattr(model.time_text_embed, "guidance_embedder") - if has_guidance_embeds and not has_guidance_embedder: - logger.warning( - "Config has guidance_embeds=True but model doesn't have guidance_embedder. " - "This may indicate a configuration mismatch." - ) - elif not has_guidance_embeds and has_guidance_embedder: - logger.warning( - "Config has guidance_embeds=False but model has guidance_embedder. " - "This may indicate a configuration mismatch." - ) - - # Note: guidance_in mappings are already included in comfyui_flux_mappings above. - # If model doesn't support guidance embeddings, the weights will be filtered out - # in _convert_comfyui_weights() based on has_guidance_embeds flag. - - param_names_mapping_fn = get_param_names_mapping(updated_mapping) - - weight_iterator = safetensors_weights_iterator(safetensors_list) - converted_weights = self._convert_comfyui_weights( - weight_iterator=weight_iterator, - qkv_size=qkv_size, - mlp_hidden_dim=mlp_hidden_dim, - has_guidance_embeds=has_guidance_embeds, - ) - - model_state_dict = model.state_dict() - missing_keys = set(model_state_dict.keys()) - unexpected_keys = [] - loaded_count = 0 - reverse_param_names_mapping = {} - - # Handle merged parameters (collect all parts before merging) - from collections import defaultdict - - to_merge_params = defaultdict(dict) - - # Process weights incrementally: load immediately after conversion - for source_name, tensor in converted_weights: - target_name, merge_index, num_params_to_merge = param_names_mapping_fn( - source_name - ) - reverse_param_names_mapping[target_name] = ( - source_name, - merge_index, - num_params_to_merge, - ) - - if merge_index is not None: - # Collect parts for merging - to_merge_params[target_name][merge_index] = tensor - if len(to_merge_params[target_name]) == num_params_to_merge: - # All parts collected, merge them - sorted_tensors = [ - to_merge_params[target_name][i] - for i in range(num_params_to_merge) - ] - merged_tensor = torch.cat(sorted_tensors, dim=0) - # Load immediately after merging - if target_name in model_state_dict: - param = model_state_dict[target_name] - loaded_tensor = merged_tensor.to( - device=param.device, dtype=param.dtype - ) - param.data.copy_(loaded_tensor) - missing_keys.discard(target_name) - loaded_count += 1 - del merged_tensor, loaded_tensor - else: - unexpected_keys.append(target_name) - # Clear merged parts - del to_merge_params[target_name] - for t in sorted_tensors: - del t - else: - # Direct mapping, load immediately - if target_name in model_state_dict: - param = model_state_dict[target_name] - # Check shape compatibility - if tensor.shape != param.shape: - logger.warning( - f"Shape mismatch for {target_name}: " - f"loaded {tensor.shape} vs model {param.shape}, skipping. " - f"Source: {source_name}" - ) - unexpected_keys.append(target_name) - del tensor - continue - - # Debug logging for norm_out.linear to verify mapping - if ( - "norm_out.linear" in target_name - or "final_layer.adaLN_modulation" in source_name - ): - logger.info( - f"Loading norm_out.linear: {source_name} -> {target_name}, " - f"shape: {tensor.shape}" - ) - - loaded_tensor = tensor.to(device=param.device, dtype=param.dtype) - param.data.copy_(loaded_tensor) - missing_keys.discard(target_name) - loaded_count += 1 - del tensor, loaded_tensor - else: - # Debug logging for unmapped parameters - if "norm_out.linear" in target_name: - logger.warning( - f"norm_out.linear parameter {target_name} not found in model state_dict. " - f"Source: {source_name}" - ) - unexpected_keys.append(target_name) - - optional_missing_keys = [] - required_missing_keys = [] - for key in missing_keys: - if key.endswith(".bias"): - # Check if corresponding weight exists (if weight exists but bias doesn't, it's optional) - weight_key = key.replace(".bias", ".weight") - if weight_key not in missing_keys: - optional_missing_keys.append(key) - else: - required_missing_keys.append(key) - else: - required_missing_keys.append(key) - - if required_missing_keys: - logger.warning( - f"Required missing keys (first 10): {required_missing_keys[:10]}..." - ) - if optional_missing_keys: - logger.info( - f"Optional missing keys (bias parameters, {len(optional_missing_keys)} total): " - f"These will use default values (zeros)" - ) - if unexpected_keys: - logger.warning(f"Unexpected keys (first 10): {unexpected_keys[:10]}...") - - logger.info(f"Successfully loaded {loaded_count} weight tensors") - - return model, reverse_param_names_mapping - - def _convert_comfyui_weights( - self, - weight_iterator: Generator[tuple[str, torch.Tensor], None, None], - qkv_size: int, - mlp_hidden_dim: int, - has_guidance_embeds: bool, - ) -> Generator[tuple[str, torch.Tensor], None, None]: - """ - Convert ComfyUI Flux weights to SGLang format. - Splits fused qkv weights into to_q/to_k/to_v plus proj_mlp. - Filters out guidance_in weights if model doesn't support guidance embeddings. - Handles scale/shift order difference between ComfyUI and AdaLayerNormContinuous. - """ - for name, tensor in weight_iterator: - if not has_guidance_embeds and name.startswith("guidance_in."): - logger.debug( - f"Skipping {name} (model doesn't support guidance embeddings)" - ) - continue - - # Split fused qkv in double blocks into separate q/k/v projections - match = re.match( - r"double_blocks\.(\d+)\.(img_attn|txt_attn)\.qkv\.(weight|bias)$", name - ) - if match: - block_idx, attn_type, param_type = match.groups() - hidden_size = qkv_size // 3 - - if tensor.shape[0] < 3 * hidden_size: - logger.warning( - f"{name} shape {tensor.shape} smaller than expected qkv size {3 * hidden_size}, skipping" - ) - continue - - if param_type == "bias": - q_tensor = tensor[:hidden_size] - k_tensor = tensor[hidden_size : 2 * hidden_size] - v_tensor = tensor[2 * hidden_size : 3 * hidden_size] - else: - q_tensor = tensor[:hidden_size, :] - k_tensor = tensor[hidden_size : 2 * hidden_size, :] - v_tensor = tensor[2 * hidden_size : 3 * hidden_size, :] - - target_prefix = f"transformer_blocks.{block_idx}.attn" - if attn_type == "img_attn": - yield f"{target_prefix}.to_q.{param_type}", q_tensor - yield f"{target_prefix}.to_k.{param_type}", k_tensor - yield f"{target_prefix}.to_v.{param_type}", v_tensor - else: - # txt_attn corresponds to encoder projections - yield f"{target_prefix}.add_q_proj.{param_type}", q_tensor - yield f"{target_prefix}.add_k_proj.{param_type}", k_tensor - yield f"{target_prefix}.add_v_proj.{param_type}", v_tensor - continue - - match = re.match(r"single_blocks\.(\d+)\.linear1\.(weight|bias)$", name) - if match: - block_idx, param_type = match.groups() - expected_size = qkv_size + mlp_hidden_dim - - if tensor.shape[0] < expected_size: - logger.warning( - f"linear1.{param_type} shape {tensor.shape} doesn't match " - f"expected size {expected_size}, skipping" - ) - continue - - # Split tensor - qkv_tensor = ( - tensor[:qkv_size] if param_type == "bias" else tensor[:qkv_size, :] - ) - mlp_tensor = ( - tensor[qkv_size:] if param_type == "bias" else tensor[qkv_size:, :] - ) - - # Split qkv into q/k/v for single blocks - hidden_size = qkv_size // 3 - if param_type == "bias": - q_tensor = qkv_tensor[:hidden_size] - k_tensor = qkv_tensor[hidden_size : 2 * hidden_size] - v_tensor = qkv_tensor[2 * hidden_size : 3 * hidden_size] - else: - q_tensor = qkv_tensor[:hidden_size, :] - k_tensor = qkv_tensor[hidden_size : 2 * hidden_size, :] - v_tensor = qkv_tensor[2 * hidden_size : 3 * hidden_size, :] - - yield ( - f"single_transformer_blocks.{block_idx}.attn.to_q.{param_type}", - q_tensor, - ) - yield ( - f"single_transformer_blocks.{block_idx}.attn.to_k.{param_type}", - k_tensor, - ) - yield ( - f"single_transformer_blocks.{block_idx}.attn.to_v.{param_type}", - v_tensor, - ) - yield ( - f"single_transformer_blocks.{block_idx}.proj_mlp.{param_type}", - mlp_tensor, - ) - elif name == "final_layer.adaLN_modulation.1.weight": - # ComfyUI: output order is [shift, scale] - # AdaLayerNormContinuous: expects [scale, shift] - # Need to swap the first half and second half of the weight matrix - # Weight shape: (2 * hidden_size, hidden_size) - # Split into two halves and swap them - half_size = tensor.shape[0] // 2 - shift_weights = tensor[:half_size, :] - scale_weights = tensor[half_size:, :] - # Swap: put scale first, then shift - swapped_tensor = torch.cat([scale_weights, shift_weights], dim=0) - logger.info( - f"Swapped scale/shift order for {name}: " - f"shape {tensor.shape} -> {swapped_tensor.shape}" - ) - yield name, swapped_tensor - elif name == "final_layer.adaLN_modulation.1.bias": - # Same swap for bias: (2 * hidden_size,) - half_size = tensor.shape[0] // 2 - shift_bias = tensor[:half_size] - scale_bias = tensor[half_size:] - swapped_tensor = torch.cat([scale_bias, shift_bias], dim=0) - logger.info( - f"Swapped scale/shift order for {name}: " - f"shape {tensor.shape} -> {swapped_tensor.shape}" - ) - yield name, swapped_tensor - else: - # Other weights pass through (handled by param_names_mapping) - yield name, tensor - - def _load_transformer_from_safetensors( - self, - server_args: ServerArgs, - loaded_modules: dict[str, torch.nn.Module] | None = None, - ) -> dict[str, Any]: - """ - Load transformer directly from safetensors file without model_index.json. - """ - if loaded_modules is not None and "transformer" in loaded_modules: - logger.info("Using provided transformer module") - components = { - "transformer": loaded_modules["transformer"], - "scheduler": self.modules.get("scheduler"), - } - return components - - if hasattr(server_args.pipeline_config, "dit_config"): - dit_config = server_args.pipeline_config.dit_config - if not isinstance(dit_config, FluxConfig): - logger.warning("dit_config is not FluxConfig, creating new FluxConfig") - dit_config = FluxConfig() - server_args.pipeline_config.dit_config = dit_config - else: - logger.info("Creating default FluxConfig") - dit_config = FluxConfig() - server_args.pipeline_config.dit_config = dit_config - - # Set guidance_embeds to True for ComfyUI Flux models - dit_config.arch_config.guidance_embeds = True - logger.info("Set guidance_embeds=True for ComfyUI Flux model") - - if dit_config.arch_config.param_names_mapping is None: - dit_config.arch_config.param_names_mapping = {} - - # ComfyUI Flux uses different parameter names than SGLang Flux - # Key differences: - # - ComfyUI: single_blocks.{i}.linear1 (fused QKV + MLP input) - # - SGLang: single_transformer_blocks.{i}.attn.to_qkv + proj_mlp (separate) - # - ComfyUI: single_blocks.{i}.linear2 - # - SGLang: single_transformer_blocks.{i}.proj_out - # - ComfyUI: double_blocks.{i}.img_attn.qkv / txt_attn.qkv - # - SGLang: transformer_blocks.{i}.attn.to_qkv / attn.to_added_qkv - - # Note: For fused layers like linear1, we need custom weight splitting logic - # which will be handled in the weight conversion function below - comfyui_flux_mappings = { - # Double stream blocks - attention layers - r"double_blocks\.(\d+)\.img_attn\.qkv\.(weight|bias)$": ( - r"transformer_blocks.\1.attn.to_qkv.\2", - None, - None, - ), - r"double_blocks\.(\d+)\.txt_attn\.qkv\.(weight|bias)$": ( - r"transformer_blocks.\1.attn.to_added_qkv.\2", - None, - None, - ), - r"double_blocks\.(\d+)\.img_attn\.proj\.(weight|bias)$": ( - r"transformer_blocks.\1.attn.to_out.0.\2", - None, - None, - ), - r"double_blocks\.(\d+)\.txt_attn\.proj\.(weight|bias)$": ( - r"transformer_blocks.\1.attn.to_add_out.\2", - None, - None, - ), - r"double_blocks\.(\d+)\.img_attn\.norm\.query_norm\.scale$": ( - r"transformer_blocks.\1.attn.norm_q.weight", - None, - None, - ), - r"double_blocks\.(\d+)\.img_attn\.norm\.key_norm\.scale$": ( - r"transformer_blocks.\1.attn.norm_k.weight", - None, - None, - ), - r"double_blocks\.(\d+)\.txt_attn\.norm\.query_norm\.scale$": ( - r"transformer_blocks.\1.attn.norm_added_q.weight", - None, - None, - ), - r"double_blocks\.(\d+)\.txt_attn\.norm\.key_norm\.scale$": ( - r"transformer_blocks.\1.attn.norm_added_k.weight", - None, - None, - ), - # Double stream blocks - MLP layers (map to net structure) - r"double_blocks\.(\d+)\.img_mlp\.0\.(weight|bias)$": ( - r"transformer_blocks.\1.ff.net.0.proj.\2", - None, - None, - ), - r"double_blocks\.(\d+)\.img_mlp\.2\.(weight|bias)$": ( - r"transformer_blocks.\1.ff.net.2.\2", - None, - None, - ), - r"double_blocks\.(\d+)\.txt_mlp\.0\.(weight|bias)$": ( - r"transformer_blocks.\1.ff_context.net.0.proj.\2", - None, - None, - ), - r"double_blocks\.(\d+)\.txt_mlp\.2\.(weight|bias)$": ( - r"transformer_blocks.\1.ff_context.net.2.\2", - None, - None, - ), - # Double stream blocks - modulation layers - r"double_blocks\.(\d+)\.img_mod\.lin\.(weight|bias)$": ( - r"transformer_blocks.\1.norm1.linear.\2", - None, - None, - ), - r"double_blocks\.(\d+)\.txt_mod\.lin\.(weight|bias)$": ( - r"transformer_blocks.\1.norm1_context.linear.\2", - None, - None, - ), - # Single stream blocks - linear2 maps to proj_out - r"single_blocks\.(\d+)\.linear2\.(weight|bias)$": ( - r"single_transformer_blocks.\1.proj_out.\2", - None, - None, - ), - # Single stream blocks - norm layers (scale -> weight) - r"single_blocks\.(\d+)\.norm\.query_norm\.scale$": ( - r"single_transformer_blocks.\1.attn.norm_q.weight", - None, - None, - ), - r"single_blocks\.(\d+)\.norm\.key_norm\.scale$": ( - r"single_transformer_blocks.\1.attn.norm_k.weight", - None, - None, - ), - # Single stream blocks - modulation (maps to norm.linear) - r"single_blocks\.(\d+)\.modulation\.lin\.(weight|bias)$": ( - r"single_transformer_blocks.\1.norm.linear.\2", - None, - None, - ), - # Time and guidance embeddings - r"^time_in\.in_layer\.(weight|bias)$": ( - r"time_text_embed.timestep_embedder.linear_1.\1", - None, - None, - ), - r"^time_in\.out_layer\.(weight|bias)$": ( - r"time_text_embed.timestep_embedder.linear_2.\1", - None, - None, - ), - r"^txt_in\.(weight|bias)$": (r"context_embedder.\1", None, None), - r"^vector_in\.in_layer\.(weight|bias)$": ( - r"time_text_embed.text_embedder.linear_1.\1", - None, - None, - ), - r"^vector_in\.out_layer\.(weight|bias)$": ( - r"time_text_embed.text_embedder.linear_2.\1", - None, - None, - ), - # Final layer mappings - r"^final_layer\.linear\.(weight|bias)$": (r"proj_out.\1", None, None), - r"^final_layer\.norm_final\.(weight|bias)$": (r"norm_out.\1", None, None), - r"^final_layer\.adaLN_modulation\.1\.(weight|bias)$": ( - r"norm_out.linear.\1", - None, - None, - ), - # Image input embedding - r"^img_in\.(weight|bias)$": (r"x_embedder.\1", None, None), - # Guidance embeddings (if model supports guidance) - r"^guidance_in\.in_layer\.(weight|bias)$": ( - r"time_text_embed.guidance_embedder.linear_1.\1", - None, - None, - ), - r"^guidance_in\.out_layer\.(weight|bias)$": ( - r"time_text_embed.guidance_embedder.linear_2.\1", - None, - None, - ), - } - - # Merge ComfyUI mappings with existing mappings (ComfyUI mappings take precedence) - updated_mapping = { - **dit_config.arch_config.param_names_mapping, - **comfyui_flux_mappings, - } - dit_config.arch_config.param_names_mapping = updated_mapping - logger.info( - "Added ComfyUI weight name mappings for Flux model. " - f"Total mappings: {len(updated_mapping)}" - ) - - cls_name = "FluxTransformer2DModel" - model_cls, _ = ModelRegistry.resolve_model_cls(cls_name) - logger.info("Resolved transformer class: %s", cls_name) - - original_mapping = None - if comfyui_flux_mappings: - original_mapping = model_cls.param_names_mapping - model_cls.param_names_mapping = updated_mapping - logger.info( - "Temporarily updated model class param_names_mapping with ComfyUI mappings. " - f"Total mappings: {len(updated_mapping)}" - ) - - safetensors_list = [self.model_path] - logger.info("Loading weights from: %s", safetensors_list) - default_dtype = resolve_precision( - server_args, "dit", precision_attr="dit_precision" - ) - server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "." - hf_config = {} - - hidden_size = ( - dit_config.arch_config.num_attention_heads - * dit_config.arch_config.attention_head_dim - ) - mlp_ratio = getattr(dit_config.arch_config, "mlp_ratio", 4.0) - mlp_hidden_dim = int(hidden_size * mlp_ratio) - qkv_size = 3 * hidden_size - has_guidance_embeds = True - - # Load and convert weights from safetensors file - model, reverse_param_names_mapping = ( - self._load_and_convert_weights_from_safetensors( - model_cls=model_cls, - dit_config=dit_config, - hf_config=hf_config, - safetensors_list=safetensors_list, - updated_mapping=updated_mapping, - qkv_size=qkv_size, - mlp_hidden_dim=mlp_hidden_dim, - has_guidance_embeds=has_guidance_embeds, - default_dtype=default_dtype, - ) - ) - - model = model.eval() - for param in model.parameters(): - param.requires_grad = False - - model.reverse_param_names_mapping = reverse_param_names_mapping - - if original_mapping is not None: - model_cls.param_names_mapping = original_mapping - - total_params = sum(p.numel() for p in model.parameters()) - logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9) - - components = { - "transformer": model, - "scheduler": self.modules.get("scheduler"), - } - - logger.info("Successfully loaded modules: %s", list(components.keys())) - return components - - def create_pipeline_stages(self, server_args: ServerArgs): - logger.info( - "ComfyUIFluxPipeline.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage" - ) - - self.add_stages( - [ - ComfyUILatentPreparationStage( - scheduler=self.get_module("scheduler"), - transformer=self.get_module("transformer"), - ), - DenoisingStage( - transformer=self.get_module("transformer"), - scheduler=self.get_module("scheduler"), - ), - ] - ) - - logger.info( - f"ComfyUIFluxPipeline stages created: {list(self._stage_name_mapping.keys())}" - ) - - -EntryClass = ComfyUIFluxPipeline diff --git a/python/sglang/multimodal_gen/runtime/pipelines/comfyui_qwen_image_pipeline.py b/python/sglang/multimodal_gen/runtime/pipelines/comfyui_qwen_image_pipeline.py deleted file mode 100644 index 95211bf3ac..0000000000 --- a/python/sglang/multimodal_gen/runtime/pipelines/comfyui_qwen_image_pipeline.py +++ /dev/null @@ -1,359 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 - -import os -from itertools import chain -from typing import Any - -import torch -from torch.distributed import init_device_mesh -from torch.distributed.fsdp import MixedPrecisionPolicy - -from sglang.multimodal_gen.configs.models.dits.qwenimage import QwenImageDitConfig -from sglang.multimodal_gen.runtime.distributed import get_local_torch_device -from sglang.multimodal_gen.runtime.loader.fsdp_load import ( - load_model_from_full_model_state_dict, - shard_model, -) -from sglang.multimodal_gen.runtime.loader.utils import ( - get_param_names_mapping, - set_default_torch_dtype, -) -from sglang.multimodal_gen.runtime.loader.weight_utils import ( - safetensors_weights_iterator, -) -from sglang.multimodal_gen.runtime.models.registry import ModelRegistry -from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import ( - ComfyUIPassThroughScheduler, -) -from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline -from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import ( - ComposedPipelineBase, -) -from sglang.multimodal_gen.runtime.pipelines_core.stages import ( - ComfyUILatentPreparationStage, - DenoisingStage, -) -from sglang.multimodal_gen.runtime.server_args import ServerArgs -from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger -from sglang.multimodal_gen.runtime.utils.precision import ( - resolve_precision, - set_mixed_precision_policy, -) - -logger = init_logger(__name__) - - -class ComfyUIQwenImagePipelineBase(LoRAPipeline, ComposedPipelineBase): - """ - Base pipeline for ComfyUI QwenImage integration with only denoising stage. - - This pipeline requires pre-processed inputs: - - prompt_embeds: Pre-encoded text embeddings (list of tensors) - - latents: Pre-processed image latents in sequence format [B, S, D] - - Usage: - generator = DiffGenerator.from_pretrained( - model_path="path/to/model", - pipeline_class_name="ComfyUIQwenImagePipeline", - device="cuda", - ) - """ - - # Subclasses should override this - zero_cond_t: bool = False - - pipeline_name = "ComfyUIQwenImagePipeline" - - _required_config_modules = [ - "transformer", - "scheduler", - ] - - def initialize_pipeline(self, server_args: ServerArgs): - """ - Initialize the pipeline with ComfyUI pass-through scheduler. - This scheduler does not modify latents, allowing ComfyUI to handle denoising. - """ - self.modules["scheduler"] = ComfyUIPassThroughScheduler( - num_train_timesteps=1000 - ) - - # Ensure VAE config is properly initialized even though we don't load the VAE model - vae_config = server_args.pipeline_config.vae_config - vae_config.post_init() - logger.info( - "Called vae_config.post_init() to set vae_scale_factor. " - f"vae_scale_factor={vae_config.arch_config.vae_scale_factor}" - ) - - def load_modules( - self, - server_args: ServerArgs, - loaded_modules: dict[str, torch.nn.Module] | None = None, - ) -> dict[str, Any]: - """ - Load modules for ComfyUIQwenImagePipeline. - - If model_path is a safetensors file, load transformer directly from it - without requiring model_index.json. Otherwise, fall back to default loading. - """ - if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"): - logger.info( - "Detected safetensors file, loading transformer directly from: %s", - self.model_path, - ) - return self._load_transformer_from_safetensors(server_args, loaded_modules) - else: - logger.info( - "Model path is a directory, using default loading method: %s", - self.model_path, - ) - return super().load_modules(server_args, loaded_modules) - - def _load_transformer_from_safetensors( - self, - server_args: ServerArgs, - loaded_modules: dict[str, torch.nn.Module] | None = None, - ) -> dict[str, Any]: - """Load transformer directly from safetensors without model_index.json.""" - - # 1) Fast path: use provided module - if loaded_modules is not None and "transformer" in loaded_modules: - logger.info("Using provided transformer module") - return { - "transformer": loaded_modules["transformer"], - "scheduler": self.modules.get("scheduler"), - } - - # 2) Build config and mappings - dit_config, updated_mapping, model_cls, default_dtype = ( - self._prepare_dit_config_and_mapping(server_args) - ) - safetensors_list = [self.model_path] - logger.info("Loading weights from: %s", safetensors_list) - - # 3) Instantiate model (meta) and optionally shard - model = self._instantiate_model( - model_cls, dit_config, default_dtype, updated_mapping, server_args - ) - - # 4) Load weights - self._load_weights_into_model( - model, safetensors_list, default_dtype, updated_mapping, server_args - ) - - components = { - "transformer": model, - "scheduler": self.modules.get("scheduler"), - } - logger.info("Successfully loaded modules: %s", list(components.keys())) - return components - - def _prepare_dit_config_and_mapping(self, server_args: ServerArgs): - from sglang.multimodal_gen.configs.models.dits.qwenimage import ( - QwenImageArchConfig, - ) - - comfyui_arch_config = QwenImageArchConfig( - patch_size=2, - in_channels=64, - out_channels=16, - num_layers=60, - attention_head_dim=128, - num_attention_heads=24, - joint_attention_dim=3584, - pooled_projection_dim=768, - guidance_embeds=False, - axes_dims_rope=(16, 56, 56), - zero_cond_t=self.zero_cond_t, - ) - dit_config = QwenImageDitConfig(arch_config=comfyui_arch_config) - server_args.pipeline_config.dit_config = dit_config - - if dit_config.arch_config.param_names_mapping is None: - dit_config.arch_config.param_names_mapping = {} - - comfyui_qwen_mappings = {r"^model\.diffusion_model\.(.*)$": r"\1"} - updated_mapping = { - **dit_config.arch_config.param_names_mapping, - **comfyui_qwen_mappings, - } - dit_config.arch_config.param_names_mapping = updated_mapping - logger.info( - "Added ComfyUI weight name mappings to param_names_mapping. " - f"Total mappings: {len(updated_mapping)}" - ) - - cls_name = "QwenImageTransformer2DModel" - model_cls, _ = ModelRegistry.resolve_model_cls(cls_name) - logger.info("Resolved transformer class: %s", cls_name) - - default_dtype = resolve_precision( - server_args, "dit", precision_attr="dit_precision" - ) - server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "." - assert server_args.hsdp_shard_dim is not None, "hsdp_shard_dim must be set" - logger.info( - "Loading %s from safetensors file, default_dtype: %s", - cls_name, - default_dtype, - ) - return dit_config, updated_mapping, model_cls, default_dtype - - def _instantiate_model( - self, - model_cls, - dit_config, - default_dtype, - updated_mapping, - server_args: ServerArgs, - ): - from sglang.multimodal_gen.runtime.platforms import current_platform - - hf_config = {} - original_mapping = model_cls.param_names_mapping - model_cls.param_names_mapping = updated_mapping - logger.info( - "Temporarily updated model class param_names_mapping with ComfyUI mappings. " - f"Total mappings: {len(updated_mapping)}" - ) - - try: - # precision-constraint: FSDP mixed precision currently uses bf16 - # parameters and fp32 reduction regardless of model load dtype. - mp_policy = MixedPrecisionPolicy( - torch.bfloat16, torch.float32, None, cast_forward_inputs=False - ) - set_mixed_precision_policy( - param_dtype=torch.bfloat16, - reduce_dtype=torch.float32, - output_dtype=None, - mp_policy=mp_policy, - ) - - with set_default_torch_dtype(default_dtype), torch.device("meta"): - model = model_cls(**{"config": dit_config, "hf_config": hf_config}) - - use_fsdp = server_args.should_use_fsdp_for_component("transformer") - if current_platform.is_mps(): - use_fsdp = False - logger.info("Disabling FSDP for MPS platform as it's not compatible") - - if use_fsdp: - device_mesh = init_device_mesh( - current_platform.device_type, - mesh_shape=( - server_args.hsdp_replicate_dim, - server_args.hsdp_shard_dim, - ), - mesh_dim_names=("replicate", "shard"), - ) - shard_model( - model, - cpu_offload=False, - reshard_after_forward=True, - mp_policy=mp_policy, - mesh=device_mesh, - fsdp_shard_conditions=getattr( - model, "_fsdp_shard_conditions", None - ), - pin_cpu_memory=server_args.pin_cpu_memory, - ) - finally: - model_cls.param_names_mapping = original_mapping - - return model - - def _load_weights_into_model( - self, - model, - safetensors_list, - default_dtype, - updated_mapping, - server_args: ServerArgs, - ): - use_fsdp = server_args.should_use_fsdp_for_component("transformer") - component_starts_on_cpu = server_args.should_start_component_on_cpu( - "transformer" - ) - # Create weight iterator for loading - weight_iterator = safetensors_weights_iterator(safetensors_list) - - # Load weights - param_names_mapping_fn = get_param_names_mapping(updated_mapping) - load_model_from_full_model_state_dict( - model, - weight_iterator, - get_local_torch_device(), - default_dtype, - strict=True, - cpu_offload=component_starts_on_cpu and not use_fsdp, - param_names_mapping=param_names_mapping_fn, - ) - - # Check for meta parameters - for n, p in chain(model.named_parameters(), model.named_buffers()): - if p.is_meta: - raise RuntimeError(f"Unexpected param or buffer {n} on meta device.") - if isinstance(p, torch.nn.Parameter): - p.requires_grad = False - - total_params = sum(p.numel() for p in model.parameters()) - logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9) - - def create_pipeline_stages(self, server_args: ServerArgs): - logger.info( - f"{self.__class__.__name__}.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage" - ) - - self.add_stages( - [ - ComfyUILatentPreparationStage( - scheduler=self.get_module("scheduler"), - transformer=self.get_module("transformer"), - ), - DenoisingStage( - transformer=self.get_module("transformer"), - scheduler=self.get_module("scheduler"), - ), - ] - ) - - logger.info( - f"{self.__class__.__name__} stages created: {list(self._stage_name_mapping.keys())}" - ) - - -class ComfyUIQwenImagePipeline(ComfyUIQwenImagePipelineBase): - """ComfyUI QwenImage pipeline for text-to-image generation.""" - - pipeline_name = "ComfyUIQwenImagePipeline" - zero_cond_t = False - - from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import ( - QwenImagePipelineConfig, - ) - from sglang.multimodal_gen.configs.sample.qwenimage import QwenImageSamplingParams - - pipeline_config_cls = QwenImagePipelineConfig - sampling_params_cls = QwenImageSamplingParams - - -class ComfyUIQwenImageEditPipeline(ComfyUIQwenImagePipelineBase): - """ComfyUI QwenImage pipeline for image-to-image editing.""" - - pipeline_name = "ComfyUIQwenImageEditPipeline" - zero_cond_t = True - - from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import ( - QwenImageEditPlusPipelineConfig, - ) - from sglang.multimodal_gen.configs.sample.qwenimage import ( - QwenImageEditPlusSamplingParams, - ) - - pipeline_config_cls = QwenImageEditPlusPipelineConfig - sampling_params_cls = QwenImageEditPlusSamplingParams - - -EntryClass = [ComfyUIQwenImagePipeline, ComfyUIQwenImageEditPipeline] diff --git a/python/sglang/multimodal_gen/runtime/pipelines/comfyui_zimage_pipeline.py b/python/sglang/multimodal_gen/runtime/pipelines/comfyui_zimage_pipeline.py deleted file mode 100644 index ac3700e7ef..0000000000 --- a/python/sglang/multimodal_gen/runtime/pipelines/comfyui_zimage_pipeline.py +++ /dev/null @@ -1,412 +0,0 @@ -# Copied and adapted from: https://github.com/hao-ai-lab/FastVideo -# SPDX-License-Identifier: Apache-2.0 - -import os -import re -from collections.abc import Generator -from itertools import chain -from typing import Any - -import torch -from torch.distributed import init_device_mesh -from torch.distributed.fsdp import MixedPrecisionPolicy - -from sglang.multimodal_gen.configs.models.dits.zimage import ZImageDitConfig -from sglang.multimodal_gen.runtime.distributed import get_local_torch_device -from sglang.multimodal_gen.runtime.loader.fsdp_load import ( - load_model_from_full_model_state_dict, - shard_model, -) -from sglang.multimodal_gen.runtime.loader.utils import ( - get_param_names_mapping, - set_default_torch_dtype, -) -from sglang.multimodal_gen.runtime.loader.weight_utils import ( - safetensors_weights_iterator, -) -from sglang.multimodal_gen.runtime.models.registry import ModelRegistry -from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import ( - ComfyUIPassThroughScheduler, -) -from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline -from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import ( - ComposedPipelineBase, -) -from sglang.multimodal_gen.runtime.pipelines_core.stages import ( - ComfyUILatentPreparationStage, - DenoisingStage, -) -from sglang.multimodal_gen.runtime.server_args import ServerArgs -from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger -from sglang.multimodal_gen.runtime.utils.precision import ( - resolve_precision, - set_mixed_precision_policy, -) - -logger = init_logger(__name__) - - -class ComfyUIZImagePipeline(LoRAPipeline, ComposedPipelineBase): - """ - Simplified pipeline for ComfyUI integration with only denoising stage. - - This pipeline requires pre-processed inputs: - - prompt_embeds: Pre-encoded text embeddings (list of tensors) - - negative_prompt_embeds: Pre-encoded negative prompt embeddings (if using CFG) - - latents: Optional initial noise latents (will be generated if not provided) - - Usage: - generator = DiffGenerator.from_pretrained( - model_path="path/to/model", - pipeline_class_name="ComfyUIZImagePipeline", - device="cuda", - ) - """ - - pipeline_name = "ComfyUIZImagePipeline" - from sglang.multimodal_gen.configs.pipeline_configs.zimage import ( - ZImagePipelineConfig, - ) - from sglang.multimodal_gen.configs.sample.zimage import ZImageSamplingParams - - pipeline_config_cls = ZImagePipelineConfig - sampling_params_cls = ZImageSamplingParams - - _required_config_modules = [ - "transformer", - "scheduler", - ] - - def initialize_pipeline(self, server_args: ServerArgs): - """ - Initialize the pipeline with ComfyUI pass-through scheduler. - This scheduler does not modify latents, allowing ComfyUI to handle denoising. - """ - self.modules["scheduler"] = ComfyUIPassThroughScheduler( - num_train_timesteps=1000 - ) - - # Ensure VAE config is properly initialized even though we don't load the VAE model - # This is necessary because get_freqs_cis uses spatial_compression_ratio - if hasattr(server_args.pipeline_config, "vae_config"): - vae_config = server_args.pipeline_config.vae_config - if hasattr(vae_config, "post_init") and not hasattr( - vae_config, "_post_init_called" - ): - vae_config.post_init() - logger.info( - "Called vae_config.post_init() to set spatial_compression_ratio. " - f"spatial_compression_ratio={vae_config.arch_config.spatial_compression_ratio}" - ) - - def load_modules( - self, - server_args: ServerArgs, - loaded_modules: dict[str, torch.nn.Module] | None = None, - ) -> dict[str, Any]: - """ - Load modules for ComfyUIZImagePipeline. - - If model_path is a safetensors file, load transformer directly from it - without requiring model_index.json. Otherwise, fall back to default loading. - """ - if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"): - logger.info( - "Detected safetensors file, loading transformer directly from: %s", - self.model_path, - ) - return self._load_transformer_from_safetensors(server_args, loaded_modules) - else: - logger.info( - "Model path is a directory, using default loading method: %s", - self.model_path, - ) - return super().load_modules(server_args, loaded_modules) - - def _convert_comfyui_qkv_weights( - self, - weight_iterator: Generator[tuple[str, torch.Tensor], None, None], - dim: int, - num_heads: int, - num_kv_heads: int, - ) -> Generator[tuple[str, torch.Tensor], None, None]: - """ - Convert ComfyUI zimage qkv weights to SGLang format. - Splits merged qkv.weight into separate to_q, to_k, to_v weights. - - Args: - weight_iterator: Iterator yielding (name, tensor) pairs from safetensors - dim: Model dimension - num_heads: Number of attention heads - num_kv_heads: Number of key-value heads - - Yields: - (name, tensor) pairs with qkv weights split into to_q, to_k, to_v - """ - head_dim = dim // num_heads - q_size = dim - k_size = head_dim * num_kv_heads - - for name, tensor in weight_iterator: - # Match qkv weights in layers, noise_refiner, or context_refiner - # Pattern: (layers|noise_refiner|context_refiner).{i}.attention.qkv.(weight|bias) - match = re.match( - r"(layers|noise_refiner|context_refiner)\.(\d+)\.attention\.qkv\.(weight|bias)$", - name, - ) - if match: - module_name, layer_idx, param_type = match.groups() - base_name = f"{module_name}.{layer_idx}.attention" - - if param_type == "weight": - # Weight shape: (q_size + k_size + v_size, dim) - # Split into q, k, v - q_weight = tensor[:q_size, :] - k_weight = tensor[q_size : q_size + k_size, :] - v_weight = tensor[q_size + k_size :, :] - - logger.debug( - f"Splitting {name} (shape {tensor.shape}) into " - f"to_q ({q_weight.shape}), to_k ({k_weight.shape}), to_v ({v_weight.shape})" - ) - - yield f"{base_name}.to_q.weight", q_weight - yield f"{base_name}.to_k.weight", k_weight - yield f"{base_name}.to_v.weight", v_weight - else: # bias - # Bias shape: (q_size + k_size + v_size,) - # Split into q, k, v - q_bias = tensor[:q_size] - k_bias = tensor[q_size : q_size + k_size] - v_bias = tensor[q_size + k_size :] - - logger.debug( - f"Splitting {name} (shape {tensor.shape}) into " - f"to_q ({q_bias.shape}), to_k ({k_bias.shape}), to_v ({v_bias.shape})" - ) - - yield f"{base_name}.to_q.bias", q_bias - yield f"{base_name}.to_k.bias", k_bias - yield f"{base_name}.to_v.bias", v_bias - else: - # Pass through other weights unchanged - yield name, tensor - - def _load_transformer_from_safetensors( - self, - server_args: ServerArgs, - loaded_modules: dict[str, torch.nn.Module] | None = None, - ) -> dict[str, Any]: - """ - Load transformer directly from safetensors file without model_index.json. - - This method: - 1. Uses hardcoded ZImageDitConfig for zimage model - 2. Loads transformer from the safetensors file - 3. Uses ComfyUIPassThroughScheduler (already created in initialize_pipeline) - """ - # Check if transformer is already provided - if loaded_modules is not None and "transformer" in loaded_modules: - logger.info("Using provided transformer module") - components = { - "transformer": loaded_modules["transformer"], - "scheduler": self.modules.get("scheduler"), - } - return components - - if hasattr(server_args.pipeline_config, "dit_config"): - dit_config = server_args.pipeline_config.dit_config - if not isinstance(dit_config, ZImageDitConfig): - logger.warning( - "dit_config is not ZImageDitConfig, creating new ZImageDitConfig" - ) - dit_config = ZImageDitConfig() - server_args.pipeline_config.dit_config = dit_config - else: - logger.info("Creating default ZImageDitConfig") - dit_config = ZImageDitConfig() - server_args.pipeline_config.dit_config = dit_config - - if dit_config.arch_config.param_names_mapping is None: - dit_config.arch_config.param_names_mapping = {} - - # Add mappings for norm layers: map from ComfyUI format (k_norm/q_norm) to SGLang format (norm_k/norm_q) - # The regex matches the source name from safetensors, and the tuple specifies the target name in the model - # Note: qkv weights are handled separately by _convert_comfyui_qkv_weights function - comfyui_norm_mappings = { - r"(.*)\.attention\.k_norm\.weight$": ( - r"\1.attention.norm_k.weight", - None, - None, - ), - r"(.*)\.attention\.q_norm\.weight$": ( - r"\1.attention.norm_q.weight", - None, - None, - ), - r"(.*)\.attention\.out\.weight$": ( - r"\1.attention.to_out.0.weight", - None, - None, - ), - r"^final_layer\.(.*)$": (r"all_final_layer.2-1.\1", None, None), - r"^x_embedder\.(.*)$": (r"all_x_embedder.2-1.\1", None, None), - } - - # Merge ComfyUI mappings with existing mappings (ComfyUI mappings take precedence) - updated_mapping = { - **dit_config.arch_config.param_names_mapping, - **comfyui_norm_mappings, - } - dit_config.arch_config.param_names_mapping = updated_mapping - logger.info( - "Added ComfyUI weight name mappings (k_norm/q_norm -> norm_k/norm_q) to param_names_mapping. " - f"Total mappings: {len(updated_mapping)}" - ) - - cls_name = "ZImageTransformer2DModel" - model_cls, _ = ModelRegistry.resolve_model_cls(cls_name) - logger.info("Resolved transformer class: %s", cls_name) - safetensors_list = [self.model_path] - logger.info("Loading weights from: %s", safetensors_list) - - default_dtype = resolve_precision( - server_args, "dit", precision_attr="dit_precision" - ) - server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "." - hf_config = {} - - assert server_args.hsdp_shard_dim is not None, "hsdp_shard_dim must be set" - logger.info( - "Loading %s from safetensors file, default_dtype: %s", - cls_name, - default_dtype, - ) - - original_mapping = model_cls.param_names_mapping - model_cls.param_names_mapping = updated_mapping - logger.info( - "Temporarily updated model class param_names_mapping with ComfyUI mappings. " - f"Total mappings: {len(updated_mapping)}" - ) - - try: - # Create model first (same as maybe_load_fsdp_model) - from sglang.multimodal_gen.runtime.platforms import current_platform - - # precision-constraint: FSDP mixed precision currently uses bf16 - # parameters and fp32 reduction regardless of model load dtype. - mp_policy = MixedPrecisionPolicy( - torch.bfloat16, torch.float32, None, cast_forward_inputs=False - ) - - set_mixed_precision_policy( - param_dtype=torch.bfloat16, - reduce_dtype=torch.float32, - output_dtype=None, - mp_policy=mp_policy, - ) - - with set_default_torch_dtype(default_dtype), torch.device("meta"): - model = model_cls(**{"config": dit_config, "hf_config": hf_config}) - - # Check if we should use FSDP - use_fsdp = server_args.should_use_fsdp_for_component("transformer") - component_starts_on_cpu = server_args.should_start_component_on_cpu( - "transformer" - ) - if current_platform.is_mps(): - use_fsdp = False - logger.info("Disabling FSDP for MPS platform as it's not compatible") - - if use_fsdp: - device_mesh = init_device_mesh( - current_platform.device_type, - mesh_shape=( - server_args.hsdp_replicate_dim, - server_args.hsdp_shard_dim, - ), - mesh_dim_names=("replicate", "shard"), - ) - shard_model( - model, - cpu_offload=False, - reshard_after_forward=True, - mp_policy=mp_policy, - mesh=device_mesh, - fsdp_shard_conditions=getattr( - model, "_fsdp_shard_conditions", None - ), - pin_cpu_memory=server_args.pin_cpu_memory, - ) - - # Get model dimensions for qkv splitting - arch_config = dit_config.arch_config - dim = arch_config.dim - num_heads = arch_config.num_attention_heads - num_kv_heads = arch_config.n_kv_heads - - # Create weight iterator with qkv conversion - base_weight_iterator = safetensors_weights_iterator(safetensors_list) - converted_weight_iterator = self._convert_comfyui_qkv_weights( - base_weight_iterator, dim, num_heads, num_kv_heads - ) - - # Load weights - param_names_mapping_fn = get_param_names_mapping(updated_mapping) - load_model_from_full_model_state_dict( - model, - converted_weight_iterator, - get_local_torch_device(), - default_dtype, - strict=True, - cpu_offload=component_starts_on_cpu and not use_fsdp, - param_names_mapping=param_names_mapping_fn, - ) - - # Check for meta parameters - for n, p in chain(model.named_parameters(), model.named_buffers()): - if p.is_meta: - raise RuntimeError( - f"Unexpected param or buffer {n} on meta device." - ) - if isinstance(p, torch.nn.Parameter): - p.requires_grad = False - finally: - model_cls.param_names_mapping = original_mapping - - total_params = sum(p.numel() for p in model.parameters()) - logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9) - - components = { - "transformer": model, - "scheduler": self.modules.get("scheduler"), - } - - logger.info("Successfully loaded modules: %s", list(components.keys())) - return components - - def create_pipeline_stages(self, server_args: ServerArgs): - logger.info( - "ComfyUIZImagePipeline.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage" - ) - - self.add_stages( - [ - ComfyUILatentPreparationStage( - scheduler=self.get_module("scheduler"), - transformer=self.get_module("transformer"), - ), - DenoisingStage( - transformer=self.get_module("transformer"), - scheduler=self.get_module("scheduler"), - ), - ] - ) - - logger.info( - f"ComfyUIZImagePipeline stages created: {list(self._stage_name_mapping.keys())}" - ) - - -EntryClass = ComfyUIZImagePipeline diff --git a/python/sglang/multimodal_gen/runtime/pipelines/flux.py b/python/sglang/multimodal_gen/runtime/pipelines/flux.py index e70a55da1c..6f3ef15ba1 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines/flux.py +++ b/python/sglang/multimodal_gen/runtime/pipelines/flux.py @@ -2,6 +2,8 @@ # SPDX-License-Identifier: Apache-2.0 +from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig +from sglang.multimodal_gen.configs.sample.flux import FluxSamplingParams from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import ( ComposedPipelineBase, @@ -32,6 +34,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs): class FluxPipeline(LoRAPipeline, ComposedPipelineBase): pipeline_name = "FluxPipeline" + # Used when the checkpoint is a single safetensors file with no + # model_index.json to derive these from. + pipeline_config_cls = FluxPipelineConfig + sampling_params_cls = FluxSamplingParams + _required_config_modules = [ "text_encoder", "text_encoder_2", diff --git a/python/sglang/multimodal_gen/runtime/pipelines/minimax_h3_pipeline.py b/python/sglang/multimodal_gen/runtime/pipelines/minimax_h3_pipeline.py index ff5ddc20bf..7f5783b67f 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines/minimax_h3_pipeline.py +++ b/python/sglang/multimodal_gen/runtime/pipelines/minimax_h3_pipeline.py @@ -190,6 +190,15 @@ class MiniMaxH3Pipeline(LoRAPipeline, ComposedPipelineBase): ) ) + def create_comfyui_stages(self, server_args: ServerArgs) -> None: + from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import ( + MiniMaxH3ComfyUIStepStage, + ) + + self.add_stage( + MiniMaxH3ComfyUIStepStage(transformer=self.get_module("transformer")) + ) + class FastH3Pipeline(MiniMaxH3Pipeline): """FastH3: 4-step DMD2-distilled MiniMax-H3 (t2va only). diff --git a/python/sglang/multimodal_gen/runtime/pipelines/qwen_image.py b/python/sglang/multimodal_gen/runtime/pipelines/qwen_image.py index 737321334c..4b6214f26c 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines/qwen_image.py +++ b/python/sglang/multimodal_gen/runtime/pipelines/qwen_image.py @@ -3,6 +3,14 @@ # SPDX-License-Identifier: Apache-2.0 from diffusers.image_processor import VaeImageProcessor +from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import ( + QwenImageEditPlusPipelineConfig, + QwenImagePipelineConfig, +) +from sglang.multimodal_gen.configs.sample.qwenimage import ( + QwenImageEditPlusSamplingParams, + QwenImageSamplingParams, +) from sglang.multimodal_gen.runtime.disaggregation.roles import RoleType from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import ( @@ -39,6 +47,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs): class QwenImagePipeline(LoRAPipeline, ComposedPipelineBase): pipeline_name = "QwenImagePipeline" + # Used when the checkpoint is a single safetensors file with no + # model_index.json to derive these from. + pipeline_config_cls = QwenImagePipelineConfig + sampling_params_cls = QwenImageSamplingParams + _required_config_modules = [ "text_encoder", "tokenizer", @@ -84,6 +97,11 @@ class QwenImageEditPipeline(LoRAPipeline, ComposedPipelineBase): class QwenImageEditPlusPipeline(QwenImageEditPipeline): pipeline_name = "QwenImageEditPlusPipeline" + # Used when the checkpoint is a single safetensors file with no + # model_index.json to derive these from. + pipeline_config_cls = QwenImageEditPlusPipelineConfig + sampling_params_cls = QwenImageEditPlusSamplingParams + def prepare_mu_layered(batch: Req, server_args: ServerArgs): base_seqlen = 256 * 256 / 16 / 16 diff --git a/python/sglang/multimodal_gen/runtime/pipelines/zimage_pipeline.py b/python/sglang/multimodal_gen/runtime/pipelines/zimage_pipeline.py index f6c92cb3fe..48636bf20c 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines/zimage_pipeline.py +++ b/python/sglang/multimodal_gen/runtime/pipelines/zimage_pipeline.py @@ -1,6 +1,8 @@ # Copied and adapted from: https://github.com/hao-ai-lab/FastVideo # SPDX-License-Identifier: Apache-2.0 +from sglang.multimodal_gen.configs.pipeline_configs.zimage import ZImagePipelineConfig +from sglang.multimodal_gen.configs.sample.zimage import ZImageSamplingParams from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline, Req from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import ( ComposedPipelineBase, @@ -27,6 +29,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs): class ZImagePipeline(LoRAPipeline, ComposedPipelineBase): pipeline_name = "ZImagePipeline" + # Used when the checkpoint is a single safetensors file with no + # model_index.json to derive these from. + pipeline_config_cls = ZImagePipelineConfig + sampling_params_cls = ZImageSamplingParams + _required_config_modules = [ "text_encoder", "tokenizer", diff --git a/python/sglang/multimodal_gen/runtime/pipelines_core/comfyui_mode.py b/python/sglang/multimodal_gen/runtime/pipelines_core/comfyui_mode.py new file mode 100644 index 0000000000..f0ea633e63 --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/pipelines_core/comfyui_mode.py @@ -0,0 +1,226 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Worker-side support for ``--comfyui-mode``. + +ComfyUI owns the sampler loop and calls SGLang once per DiT step. This +module covers both halves of that contract: + +- pipeline assembly: drop every module except the transformer, install a + pass-through scheduler, keep only the stages needed for one forward +- run cache: after the first step, text embeddings and packed extras stay + on the worker; later steps send latents and the timestep + +Single-file / GGUF DiT loading lives in +``sglang.multimodal_gen.runtime.loader.comfyui_checkpoints``. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any, Callable + +from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import ( + ComfyUIPassThroughScheduler, +) +from sglang.multimodal_gen.runtime.server_args import ServerArgs + +if TYPE_CHECKING: + from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import ( + ComposedPipelineBase, + ) + +COMFYUI_REQUIRED_MODULES = ["transformer", "scheduler"] + +_CONDITIONING_FIELDS = ( + "prompt_embeds", + "negative_prompt_embeds", + "prompt_seq_lens", + "negative_prompt_seq_lens", + "pooled_embeds", + "neg_pooled_embeds", + "image_latent", + "vae_image_sizes", + "prompt_attention_mask", + "negative_attention_mask", + "prompt_embeds_mask", + "negative_prompt_embeds_mask", + "sigmas", +) +_SESSION_SKIP_EXTRA = frozenset( + {"comfyui_session_id", "comfyui_cond_key", "comfyui_cache_fp"} +) +_SESSIONS: dict[str, dict[str, Any]] = {} +_RUNS: dict[str, Any] = {} + + +def is_comfyui_mode(server_args: ServerArgs) -> bool: + return bool(server_args.comfyui_mode) + + +def initialize_comfyui_pipeline( + pipeline: ComposedPipelineBase, server_args: ServerArgs +) -> None: + """Install the pass-through scheduler and finish deriving VAE geometry. + + The VAE model itself is never loaded, but its config still carries the + compression ratios that RoPE frequency construction reads. + """ + pipeline.modules["scheduler"] = ComfyUIPassThroughScheduler( + num_train_timesteps=1000 + ) + + vae_config = getattr(server_args.pipeline_config, "vae_config", None) + if ( + vae_config is not None + and hasattr(vae_config, "post_init") + and not hasattr(vae_config, "_post_init_called") + ): + # Flux / Qwen RoPE reads arch_config.vae_scale_factor, which + # post_init() derives from dim_mult. Do not skip just because a + # ComfyUI DiT checkpoint has no latents_mean — that field is only + # required when this process owns the VAE (MiniMax-H3 native). + vae_config.post_init() + + +def create_comfyui_pipeline_stages( + pipeline: ComposedPipelineBase, server_args: ServerArgs +) -> None: + if hasattr(pipeline, "create_comfyui_stages"): + pipeline.create_comfyui_stages(server_args) + return + + from sglang.multimodal_gen.runtime.pipelines_core.stages import ( + ComfyUILatentPreparationStage, + DenoisingStage, + ) + + transformer = pipeline.get_module("transformer") + scheduler = pipeline.get_module("scheduler") + pipeline.add_stages( + [ + ComfyUILatentPreparationStage(scheduler=scheduler, transformer=transformer), + DenoisingStage(transformer=transformer, scheduler=scheduler), + ] + ) + + +def session_id_from_req(req) -> str | None: + extra = getattr(req, "extra", None) or {} + sid = extra.get("comfyui_session_id") + return sid if sid else None + + +def _session_family(session_id: str) -> str: + return session_id.split(":", 1)[0] + + +def _evict_other_runs(session_id: str) -> None: + """Drop older ``{uuid}:{run}`` slots for this executor.""" + prefix = _session_family(session_id) + ":" + for store in (_SESSIONS, _RUNS): + for key in list(store): + if key.startswith(prefix) and key != session_id: + store.pop(key, None) + + +def bind_comfyui_session(req): + """Restore cached conditioning, then refresh the cache from whatever is set. + + Conditioning is stored per ``comfyui_cond_key`` so CFG pos/neg do not + overwrite each other. Packed extras are restored only when + ``comfyui_cache_fp`` matches; a mismatch must not revive a stale layout. + """ + sid = session_id_from_req(req) + if not sid: + return req + + _evict_other_runs(sid) + extra = dict(getattr(req, "extra", None) or {}) + cond_key = extra.get("comfyui_cond_key") or "_" + cached = _SESSIONS.get(sid) or {} + cond_snap = (cached.get("_conds") or {}).get(cond_key) or {} + current_fp = extra.get("comfyui_cache_fp") + cached_fp = cached.get("_fp") + extra_ok = current_fp is None or cached_fp is None or current_fp == cached_fp + + for name, value in cond_snap.items(): + if _is_empty(getattr(req, name, None)): + setattr(req, name, value) + if extra_ok: + for key, value in (cached.get("_extra") or {}).items(): + if _is_empty(extra.get(key)): + extra[key] = value + req.extra = extra + + new_cond = {} + for name in _CONDITIONING_FIELDS: + value = getattr(req, name, None) + if not _is_empty(value): + new_cond[name] = value + extra = getattr(req, "extra", None) or {} + extra_snapshot = {} + for key, value in extra.items(): + if key in _SESSION_SKIP_EXTRA or _is_empty(value): + continue + extra_snapshot[key] = value + conds = dict(cached.get("_conds") or {}) + if new_cond: + conds[cond_key] = new_cond + snapshot: dict[str, Any] = {"_conds": conds} + if extra_snapshot: + snapshot["_extra"] = extra_snapshot + if current_fp is not None: + snapshot["_fp"] = current_fp + elif extra_ok and cached_fp is not None: + snapshot["_fp"] = cached_fp + if conds or extra_snapshot: + _SESSIONS[sid] = snapshot + return req + + +def get_run_state(req): + sid = session_id_from_req(req) + if sid: + return _RUNS.get(sid) + return None + + +def set_run_state(req, state: Any) -> None: + sid = session_id_from_req(req) + if sid: + _RUNS[sid] = state + + +def pop_run_state(req): + sid = session_id_from_req(req) + if sid: + return _RUNS.pop(sid, None) + return None + + +def get_or_create_run_state(req, factory: Callable[[], Any]): + """Session-scoped opaque state. Created once per run.""" + existing = get_run_state(req) + if existing is not None: + return existing + state = factory() + set_run_state(req, state) + return state + + +def release_run_state(session_id: str | None) -> None: + """Drop per-run objects; keep cached conditioning for the next step.""" + if session_id: + _RUNS.pop(session_id, None) + + +def release_comfyui_session(session_id: str | None) -> None: + if session_id: + _SESSIONS.pop(session_id, None) + _RUNS.pop(session_id, None) + + +def _is_empty(value) -> bool: + if value is None: + return True + if isinstance(value, (list, tuple)): + return len(value) == 0 + return False diff --git a/python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py b/python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py index 296d453d77..352cf45cb8 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py +++ b/python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py @@ -18,6 +18,10 @@ from sglang.multimodal_gen.runtime.disaggregation.roles import ( RoleType, filter_modules_for_role, ) +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( + is_comfyui_single_file, + load_comfyui_transformer, +) from sglang.multimodal_gen.runtime.loader.component_loaders.component_loader import ( ComponentLoader, PipelineComponentLoader, @@ -32,6 +36,12 @@ from sglang.multimodal_gen.runtime.managers.memory_managers.component_manager im ComponentResidencyStrategy, get_global_component_residency_manager, ) +from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import ( + COMFYUI_REQUIRED_MODULES, + create_comfyui_pipeline_stages, + initialize_comfyui_pipeline, + is_comfyui_mode, +) from sglang.multimodal_gen.runtime.pipelines_core.executors.pipeline_executor import ( PipelineExecutor, ) @@ -123,6 +133,9 @@ class ComposedPipelineBase(ABC): ) if base_required_config_modules is None: raise NotImplementedError("Subclass must set _required_config_modules") + if is_comfyui_mode(server_args): + # ComfyUI owns text encoding, VAE decode, and the sampler loop. + base_required_config_modules = COMFYUI_REQUIRED_MODULES self._unfiltered_required_config_modules = tuple(base_required_config_modules) self._required_config_modules = list(self._unfiltered_required_config_modules) self._extra_config_module_map = dict(self._extra_config_module_map) @@ -166,6 +179,12 @@ class ComposedPipelineBase(ABC): def __post_init__(self) -> None: assert self.server_args is not None, "server_args must be set" + if is_comfyui_mode(self.server_args): + initialize_comfyui_pipeline(self, self.server_args) + logger.info("Creating ComfyUI pipeline stages...") + create_comfyui_pipeline_stages(self, self.server_args) + return + self.initialize_pipeline(self.server_args) logger.info("Creating pipeline stages...") @@ -425,6 +444,8 @@ class ComposedPipelineBase(ABC): loaded_modules: Optional[Dict[str, torch.nn.Module]] = None, If provided, loaded_modules will be used instead of loading from config/pretrained weights. """ + if is_comfyui_mode(server_args) and is_comfyui_single_file(self.model_path): + return load_comfyui_transformer(self, server_args, loaded_modules) model_index = self._load_config() logger.info("Loading pipeline modules from config: %s", model_index) diff --git a/python/sglang/multimodal_gen/runtime/pipelines_core/schedule_batch.py b/python/sglang/multimodal_gen/runtime/pipelines_core/schedule_batch.py index d931264e12..5e07f55b23 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines_core/schedule_batch.py +++ b/python/sglang/multimodal_gen/runtime/pipelines_core/schedule_batch.py @@ -155,7 +155,7 @@ class Req: raw_latent_shape: torch.Tensor | None = None did_sp_shard_latents: bool = False sp_video_start_frame: int = 0 - noise_pred: torch.Tensor | None = None + noise_pred: torch.Tensor | list | tuple | None = None # vae-encoded condition image image_latent: torch.Tensor | list[torch.Tensor] | None = None condition_image_latent_ids: torch.Tensor | list[torch.Tensor] | None = None @@ -502,7 +502,7 @@ class OutputBatch: metrics_list: Optional[list[Optional[RequestMetrics]]] = None # For ComfyUI integration: noise prediction from denoising stage - noise_pred: torch.Tensor | None = None + noise_pred: torch.Tensor | list | tuple | None = None peak_memory_mb: float = 0.0 usage: dict[str, Any] | None = None diff --git a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/comfyui_latent_preparation.py b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/comfyui_latent_preparation.py index 0b4e53a928..b273576583 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/comfyui_latent_preparation.py +++ b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/comfyui_latent_preparation.py @@ -1,114 +1,42 @@ # SPDX-License-Identifier: Apache-2.0 -""" -ComfyUI latent preparation stage with device mismatch fix. -This stage extends LatentPreparationStage to handle device mismatch issues -that occur when tensors are pickled and unpickled via broadcast_pyobj in -multi-GPU scenarios. +"""ComfyUI latent prep: restore the worker session and bind the pass-through scheduler. + +Multi-rank hops move CUDA tensors with NCCL, so this stage no longer walks +every field to fix pickle/gloo device mismatches. """ -import dataclasses - -import torch - -from sglang.multimodal_gen.runtime.distributed import ( - get_local_torch_device, - get_sp_group, +from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import ( + bind_comfyui_session, +) +from sglang.multimodal_gen.runtime.pipelines_core.diffusion_scheduler_utils import ( + get_or_create_request_scheduler, ) -from sglang.multimodal_gen.runtime.distributed.parallel_state import get_sp_world_size from sglang.multimodal_gen.runtime.pipelines_core.stages.latent_preparation import ( LatentPreparationStage, ) -from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger - -logger = init_logger(__name__) class ComfyUILatentPreparationStage(LatentPreparationStage): - """ - ComfyUI-specific latent preparation stage with device mismatch fix. + """One DiT step: restore cached conditioning, then prepare latents.""" - This stage extends LatentPreparationStage to automatically fix device - mismatches for tensor fields on non-source ranks in multi-GPU scenarios. - """ - - @staticmethod - def _fix_tensor_device(value, target_device): - """Recursively fix tensor device, handling single tensors, lists, and tuples.""" - if isinstance(value, torch.Tensor): - if value.device != target_device: - return value.detach().clone().to(target_device) - return value - elif isinstance(value, list): - return [ - ComfyUILatentPreparationStage._fix_tensor_device(v, target_device) - for v in value - ] - elif isinstance(value, tuple): - return tuple( - ComfyUILatentPreparationStage._fix_tensor_device(v, target_device) - for v in value - ) - return value - - @staticmethod - def _has_tensor(value): - """Check if value contains any tensor.""" - if isinstance(value, torch.Tensor): - return True - elif isinstance(value, (list, tuple)): - return any(ComfyUILatentPreparationStage._has_tensor(v) for v in value) - return False + def verify_input(self, batch, server_args): + bind_comfyui_session(batch) + return super().verify_input(batch, server_args) def forward(self, batch, server_args): - """ - Prepare latents with device mismatch fix for ComfyUI pipelines. - - This method first fixes device mismatches for all tensor fields, - then calls the parent class's forward method, and ensures raw_latent_shape - is set correctly (before packing, for proper unpadding later). - """ - # Fix device mismatch for tensor fields on non-source ranks - if get_sp_world_size() > 1: - sp_group = get_sp_group() - target_device = get_local_torch_device() - - if sp_group.rank != 0: - logger.debug( - f"[ComfyUILatentPreparationStage] Fixing tensor device on rank={sp_group.rank} " - f"target_device={target_device}" - ) - - if dataclasses.is_dataclass(batch): - for field in dataclasses.fields(batch): - value = getattr(batch, field.name, None) - if value is not None and self._has_tensor(value): - fixed_value = self._fix_tensor_device(value, target_device) - setattr(batch, field.name, fixed_value) - else: - for attr_name in dir(batch): - if not attr_name.startswith("_") and not callable( - getattr(batch, attr_name, None) - ): - try: - value = getattr(batch, attr_name, None) - if value is not None and self._has_tensor(value): - fixed_value = self._fix_tensor_device( - value, target_device - ) - setattr(batch, attr_name, fixed_value) - except (AttributeError, TypeError): - continue + # DenoisingStage reads batch.scheduler. Native pipelines attach it in + # TimestepPreparationStage; ComfyUI already owns the timestep schedule. + get_or_create_request_scheduler(batch, self.scheduler) original_latents_shape = None if batch.latents is not None: original_latents_shape = batch.latents.shape - # Call parent class's forward method result = super().forward(batch, server_args) if original_latents_shape is not None: - # Preserve the original shape before any potential packing/conversion - # (e.g., 4D spatial -> 3D sequence) to ensure proper unpadding later. + # Preserve the original shape before any packing/conversion + # (e.g., 4D spatial -> 3D sequence) so unpadding stays correct. result.raw_latent_shape = original_latents_shape return result diff --git a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/__init__.py b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/__init__.py index a85586468d..8b93c5202a 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/__init__.py +++ b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/__init__.py @@ -3,6 +3,7 @@ """MiniMax H3-specific pipeline stages.""" from .stages.audio_encoding import MiniMaxH3AudioEncodingStage +from .stages.comfyui_step import MiniMaxH3ComfyUIStepStage from .stages.decoding import MiniMaxH3DecodingStage from .stages.denoising import MiniMaxH3DenoisingStage from .stages.latent_preparation import MiniMaxH3LatentPreparationStage @@ -12,6 +13,7 @@ from .stages.visual_encoding import MiniMaxH3VisualEncodingStage __all__ = [ "MiniMaxH3AudioEncodingStage", + "MiniMaxH3ComfyUIStepStage", "MiniMaxH3DecodingStage", "MiniMaxH3DenoisingStage", "MiniMaxH3LatentPreparationStage", diff --git a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/comfyui_step.py b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/comfyui_step.py new file mode 100644 index 0000000000..dac51f2365 --- /dev/null +++ b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/minimax_h3/stages/comfyui_step.py @@ -0,0 +1,799 @@ +# SPDX-License-Identifier: Apache-2.0 +"""ComfyUI per-step DiT stage for MiniMax H3. + +ComfyUI owns the sampler loop. This stage runs one ``MiniMaxH3DiTModel.forward`` +via ``MiniMaxH3DenoiseBranch.forward_kwargs`` and writes unpacked +``[video, audio]`` velocities onto ``batch.noise_pred``. +""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass +from typing import Any + +import torch + +_logger = logging.getLogger(__name__) + +from sglang.multimodal_gen.configs.models.dits.minimax_h3 import ( + MINIMAX_H3_ADALN_MODALITY_NUM, + MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT, +) +from sglang.multimodal_gen.runtime.models.dits.minimax_h3 import ( + _FORWARD_SUPPORTED_KWARGS, +) +from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import ( + bind_comfyui_session, + get_or_create_run_state, + get_run_state, + pop_run_state, + release_run_state, + session_id_from_req, +) +from sglang.multimodal_gen.runtime.pipelines_core.schedule_batch import Req +from sglang.multimodal_gen.runtime.pipelines_core.stages.base import PipelineStage +from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.denoise_loop import ( + MINIMAX_H3_AUDIO_REF_COND_TIMESTEP, + MINIMAX_H3_IMGVID_COND_TIMESTEP, + MiniMaxH3DenoiseBranch, +) +from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.packed_sequence import ( + minimax_h3_packed_sequence, + minimax_h3_packed_sequence_ref2va_blocks, +) +from sglang.multimodal_gen.runtime.server_args import ServerArgs + +DEFAULT_SIGMA_SHIFT_VIDEO = 12.0 +DEFAULT_SIGMA_SHIFT_AUDIO = 3.0 +_PATCH_SIZE = (1, 2, 2) +_SEG_TOKEN_TAG = { + "text": 1, + "video": 0, + "cond": 0, + "ref_img": 0, + "audio": 2, + "cond_audio": 2, + "ref_audio": 2, +} + + +def time_shift_sigma(sigma, from_shift, to_shift): + base = sigma / (from_shift + sigma * (1.0 - from_shift)) + return to_shift * base / (1.0 + (to_shift - 1.0) * base) + + +def time_shift_slope(sigma, from_shift, to_shift): + """d(sigma_to)/d(sigma_from) at the same base-grid point.""" + base = sigma / (from_shift + sigma * (1.0 - from_shift)) + return (to_shift * (1.0 + (from_shift - 1.0) * base) ** 2) / ( + from_shift * (1.0 + (to_shift - 1.0) * base) ** 2 + ) + + +def patchify_video(latent, patch_size=_PATCH_SIZE): + b, c, t_full, h_full, w_full = latent.shape + pt, ph, pw = patch_size + t, h, w = t_full // pt, h_full // ph, w_full // pw + x = latent.reshape(b, c, t, pt, h, ph, w, pw) + x = torch.einsum("nctrhpwq->nthwcrpq", x) + return x.reshape(b * t * h * w, c * pt * ph * pw) + + +def unpatchify_video(rows, t, h, w, c=24, patch_size=_PATCH_SIZE): + pt, ph, pw = patch_size + x = rows.reshape(-1, t, h, w, c, pt, ph, pw) + x = torch.einsum("nthwcrpq->nctrhpwq", x) + return x.reshape(-1, c, t * pt, h * ph, w * pw) + + +def pack_audio(latent): + _b, _c, ch, t = latent.shape + return latent[0].permute(1, 2, 0).reshape(ch * t, latent.shape[1]) + + +def unpack_audio(rows, ch=2): + t = rows.shape[0] // ch + return rows.reshape(ch, t, rows.shape[-1]).permute(2, 0, 1).unsqueeze(0) + + +def pad_to_patch_size(video, patch_size=_PATCH_SIZE): + pt, ph, pw = patch_size + t, h, w = video.shape[2], video.shape[3], video.shape[4] + pad_t = (pt - t % pt) % pt + pad_h = (ph - h % ph) % ph + pad_w = (pw - w % pw) % pw + if pad_t == pad_h == pad_w == 0: + return video + return torch.nn.functional.pad(video, (0, pad_w, 0, pad_h, 0, pad_t)) + + +def _as_text_embeddings(context: torch.Tensor) -> torch.Tensor: + if context.ndim == 3: + return context[0] + return context + + +def _payload_has_keyframes(payload: dict[str, Any] | None) -> bool: + return bool((payload or {}).get("keyframes")) + + +def _payload_refs(payload: dict[str, Any] | None) -> list[dict[str, Any]]: + refs = (payload or {}).get("refs") or [] + if not isinstance(refs, (list, tuple)): + raise TypeError(f"minimax_payload.refs must be a sequence, got {type(refs)!r}") + return list(refs) + + +def _apply_cond_noise(rows: torch.Tensor, aug: float, seed: int) -> torch.Tensor: + if aug >= 1.0: + return rows + gen = torch.Generator("cpu").manual_seed(int(seed)) + noise = torch.randn(rows.shape, generator=gen, dtype=torch.float32) + return aug * rows + (1.0 - aug) * noise.to(rows.device) + + +def _ref_blocks_for_packed(refs: list[dict[str, Any]]) -> list[dict[str, Any]]: + blocks: list[dict[str, Any]] = [] + for ref in refs: + block = { + key: value + for key, value in ref.items() + if key not in {"latent", "audio_latent"} + } + if "kind" not in block and "type" in block: + block["kind"] = block["type"] + blocks.append(block) + return blocks + + +def serialize_comfyui_layout(layout: Any) -> dict[str, Any] | None: + """Turn a ComfyUI ``PackedLayout`` into pickle/IPC-safe tensors.""" + required = ( + "seq_len", + "signature", + "segments", + "position_ids", + "img_pos", + "img_update", + "audio_pos", + "audio_update", + ) + if layout is None or any(not hasattr(layout, name) for name in required): + return None + return { + "seq_len": int(layout.seq_len), + "signature": tuple(int(value) for value in layout.signature), + "segments": [ + (int(start), int(stop), str(kind)) for start, stop, kind in layout.segments + ], + "position_ids": layout.position_ids.detach().cpu().to(torch.float64), + "img_pos": layout.img_pos.detach().cpu().to(torch.long), + "img_update": layout.img_update.detach().cpu().to(torch.bool), + "audio_pos": layout.audio_pos.detach().cpu().to(torch.long), + "audio_update": layout.audio_update.detach().cpu().to(torch.bool), + } + + +def _layout_signature_matches( + layout: dict[str, Any], + video_x: torch.Tensor, + audio_x: torch.Tensor, + text: torch.Tensor, +) -> bool: + """ComfyUI ``PackedLayout.signature`` is ``(text, T, H, W, audio_t)``. + + A cached layout from an earlier sampler pass (e.g. pre-upscale 14×8 + tokens) must not be applied to a different spatial grid. + """ + sig = layout.get("signature") + if sig is None: + return False + values = tuple(int(v) for v in sig[:5]) + if len(values) < 5: + return False + text_len, latent_t, latent_h, latent_w, audio_t = values + return ( + text_len == int(text.shape[0]) + and latent_t == int(video_x.shape[2]) + and latent_h == int(video_x.shape[3]) + and latent_w == int(video_x.shape[4]) + and audio_t == int(audio_x.shape[-1]) + ) + + +def comfyui_layout_to_packed(layout: dict[str, Any]) -> dict[str, Any]: + """Rebuild the SGLD packed dict from a serialized ComfyUI layout. + + Only pads ``seq_len`` to 64. Row order, RoPE ids, and update masks stay + exactly as ComfyUI computed them. + """ + used = int(layout["seq_len"]) + seq_len = ( + (used + MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT - 1) + // MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT + * MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT + ) + position_ids = layout["position_ids"].to(dtype=torch.float64) + grid = torch.zeros(seq_len, 3, dtype=torch.float64) + grid[:used] = position_ids[:used] + token_tags = torch.full((seq_len,), -1, dtype=torch.long) + text_pos = torch.empty(0, dtype=torch.long) + for start, stop, kind in layout["segments"]: + token_tags[start:stop] = _SEG_TOKEN_TAG.get(kind, -1) + if kind == "text": + text_pos = torch.arange(start, stop, dtype=torch.long) + return { + "seq_len": seq_len, + "img_pos": layout["img_pos"].view(-1).to(torch.long), + "audio_pos": layout["audio_pos"].view(-1).to(torch.long), + "text_pos": text_pos, + "update_mask": layout["img_update"].view(-1).to(torch.bool), + "audio_update_mask": layout["audio_update"].view(-1).to(torch.bool), + "img_position_ids": grid, + "token_tags": token_tags, + "cu_seqlens": torch.tensor([0, used, seq_len], dtype=torch.int32), + } + + +def _latent_items(payload: dict[str, Any], key: str) -> list[Any]: + items = payload.get(key) or [] + if not isinstance(items, (list, tuple)): + return [items] + return [item for item in items if item is not None] + + +def _assemble_media_rows( + video: torch.Tensor, + audio: torch.Tensor, + payload: dict[str, Any], +) -> tuple[torch.Tensor, torch.Tensor]: + video_target = patchify_video(video, _PATCH_SIZE) + audio_target = pack_audio(audio) + vis_aug = float( + payload.get("visual_cond_noise_aug", MINIMAX_H3_IMGVID_COND_TIMESTEP) + ) + aud_aug = float( + payload.get("audio_cond_noise_aug", MINIMAX_H3_AUDIO_REF_COND_TIMESTEP) + ) + seed = int(payload.get("seed", 0)) + + cond_video_src = list(_latent_items(payload, "cond_video_latents")) + cond_audio_src = list(_latent_items(payload, "cond_audio_latents")) + if not cond_video_src and not cond_audio_src: + for block in list(payload.get("keyframes") or []) + _payload_refs(payload): + if block.get("latent") is not None: + cond_video_src.append(block["latent"]) + if block.get("audio_latent") is not None: + cond_audio_src.append(block["audio_latent"]) + + cond_video: list[torch.Tensor] = [] + for latent in cond_video_src: + rows = patchify_video( + pad_to_patch_size( + latent.to(device=video.device, dtype=torch.float32), + _PATCH_SIZE, + ), + _PATCH_SIZE, + ) + cond_video.append(_apply_cond_noise(rows, vis_aug, seed)) + cond_audio: list[torch.Tensor] = [] + for latent in cond_audio_src: + rows = pack_audio(latent.to(device=audio.device, dtype=torch.float32)) + cond_audio.append(_apply_cond_noise(rows, aud_aug, seed + 1)) + + video_rows = ( + torch.cat(cond_video + [video_target], dim=0) if cond_video else video_target + ) + audio_rows = ( + torch.cat(cond_audio + [audio_target], dim=0) if cond_audio else audio_target + ) + return video_rows, audio_rows + + +def _token_tags_for_payload( + packed: dict[str, torch.Tensor], + payload: dict[str, Any], +) -> torch.Tensor: + tags = packed["token_tags"] + text_tags = payload.get("text_token_tags") + if text_tags is None: + return tags + text_pos = packed["text_pos"].view(-1) + text_tags = text_tags.reshape(-1).to(dtype=torch.long) + if int(text_tags.numel()) != int(text_pos.numel()): + raise ValueError( + "text_token_tags length " + f"{int(text_tags.numel())} != packed text_len {int(text_pos.numel())}" + ) + tags = tags.clone() + tags[text_pos] = text_tags + return tags + + +def _sigma_list(sample_sigmas: Any) -> list[float] | None: + n = _sample_sigmas_len(sample_sigmas) + if n is None: + return None + if torch.is_tensor(sample_sigmas): + return [float(v) for v in sample_sigmas.reshape(-1).tolist()] + return [float(v) for v in sample_sigmas] + + +def _as_spatial_mask(mask: Any) -> torch.Tensor | None: + if mask is None: + return None + values = mask + if not torch.is_tensor(values): + values = torch.as_tensor(values) + if int(values.numel()) == 0: + return None + if values.ndim == 5: + values = values[0, 0] + elif values.ndim == 4: + values = values[0] + elif values.ndim == 2: + values = values.unsqueeze(0) + return values.to(dtype=torch.float32) + + +def mask_row_values(mask: Any, latent_t: int, lat_h: int, lat_w: int): + """Match ComfyUI ``mask_row_values``: [T,H,W] → one float per 2×2 patch row.""" + values = _as_spatial_mask(mask) + if values is None: + return None + if values.ndim != 3: + raise ValueError( + f"denoise_mask must be [T,H,W] after squeeze, got {tuple(values.shape)}" + ) + if values.shape[0] == 1 and latent_t > 1: + values = values.expand(latent_t, *values.shape[1:]) + elif values.shape[0] != latent_t: + raise ValueError( + f"denoise_mask T={int(values.shape[0])} != padded latent T={latent_t}" + ) + pad_w = lat_w - int(values.shape[-1]) + pad_h = lat_h - int(values.shape[-2]) + if pad_w > 0 or pad_h > 0: + values = torch.nn.functional.pad( + values, (0, max(pad_w, 0), 0, max(pad_h, 0)), mode="replicate" + ) + values = values[:latent_t, :lat_h, :lat_w] + values = values.reshape(latent_t, lat_h // 2, 2, lat_w // 2, 2).amax(dim=(2, 4)) + rows = values.reshape(-1) + if bool((rows >= 1.0 - 1e-3).all()): + return None + return rows + + +def _audio_mask_values(mask: Any): + if mask is None: + return None + values = mask.reshape(-1).to(dtype=torch.float32) + if int(values.numel()) == 0: + return None + if bool((values >= 1.0 - 1e-3).all()): + return None + return values + + +def _overlay_per_row_timesteps( + fk: dict[str, Any], + branch: MiniMaxH3DenoiseBranch, + video_rows_t: torch.Tensor | None, + audio_rows_t: torch.Tensor | None, +) -> dict[str, Any]: + """Apply ComfyUI per-row denoise_mask only on target video/audio rows.""" + if video_rows_t is None and audio_rows_t is None: + return fk + unique = fk["unique_timesteps"].to(torch.float32) + inverse = fk["inverse_indices"].clone() + device = inverse.device + extras = [unique.reshape(-1).detach().cpu()] + if video_rows_t is not None: + extras.append(video_rows_t.reshape(-1).to(torch.float32).cpu()) + if audio_rows_t is not None: + extras.append(audio_rows_t.reshape(-1).to(torch.float32).cpu()) + new_unique = torch.unique(torch.cat(extras), sorted=True) + inverse = torch.searchsorted(new_unique, unique.cpu()).to(device)[inverse] + if video_rows_t is not None: + mapped = torch.searchsorted( + new_unique, video_rows_t.reshape(-1).to(torch.float32).cpu() + ) + tgt = branch.img_target_seq_idx + if int(mapped.numel()) != int(tgt.numel()): + raise ValueError( + "denoise_mask rows " + f"{int(mapped.numel())} != video target rows {int(tgt.numel())}" + ) + inverse[tgt] = mapped.to(device) + if audio_rows_t is not None: + mapped = torch.searchsorted( + new_unique, audio_rows_t.reshape(-1).to(torch.float32).cpu() + ) + tgt = branch.audio_target_seq_idx + if int(mapped.numel()) != int(tgt.numel()): + raise ValueError( + "audio_denoise_mask rows " + f"{int(mapped.numel())} != audio target rows {int(tgt.numel())}" + ) + inverse[tgt] = mapped.to(device) + fk["unique_timesteps"] = new_unique.to(device=device, dtype=unique.dtype) + fk["inverse_indices"] = inverse + fk["block_combined_indices"] = ( + branch.block_token_tags + + inverse[branch.local_row_slice] * MINIMAX_H3_ADALN_MODALITY_NUM + ) + return fk + + +def comfyui_payload_to_branch_inputs( + video_x: torch.Tensor, + audio_x: torch.Tensor, + context: torch.Tensor, + minimax_payload: dict[str, Any] | None, + sample_sigmas: Any, + timestep: torch.Tensor, + transformer_options: dict[str, Any] | None = None, + layout: dict[str, Any] | None = None, + denoise_mask: Any = None, + audio_denoise_mask: Any = None, +) -> dict[str, Any]: + """Translate one ComfyUI H3 step into Branch + ``forward_kwargs`` inputs. + + ``context`` is raw Qwen states (last dim 5120). Token refinement is + precomputed once on the worker after the Branch is created. + """ + payload = minimax_payload or {} + orig_shape = tuple(video_x.shape) + video = pad_to_patch_size(video_x.to(torch.float32), _PATCH_SIZE) + audio = audio_x.to(torch.float32) + text = _as_text_embeddings(context) + if layout is not None and not _layout_signature_matches( + layout, video_x, audio, text + ): + _logger.warning( + "Ignoring stale ComfyUI PackedLayout signature %s for video %s " + "audio %s text_len %s", + layout.get("signature"), + tuple(video_x.shape), + tuple(audio.shape), + int(text.shape[0]), + ) + layout = None + if layout is None and _payload_has_keyframes(payload): + raise NotImplementedError( + "MiniMaxH3 ComfyUI step needs a serialized PackedLayout for fl2va keyframes" + ) + text_len = int(text.shape[0]) + latent_t, latent_h, latent_w = ( + int(video.shape[2]), + int(video.shape[3]), + int(video.shape[4]), + ) + audio_t = int(audio.shape[-1]) + refs = _payload_refs(payload) + if layout is not None: + packed = comfyui_layout_to_packed(layout) + elif refs: + packed = minimax_h3_packed_sequence_ref2va_blocks( + text_len=text_len, + latent_t=latent_t, + latent_h=latent_h, + latent_w=latent_w, + audio_t=audio_t, + ref_blocks=_ref_blocks_for_packed(refs), + ) + else: + packed = minimax_h3_packed_sequence( + text_len=text_len, + latent_t=latent_t, + latent_h=latent_h, + latent_w=latent_w, + audio_t=audio_t, + include_keyframe_cond=False, + ) + used = int(packed["cu_seqlens"][1]) + video_rows, audio_rows = _assemble_media_rows(video, audio, payload) + + opts = transformer_options or {} + shift_v = float(opts.get("minimax_h3_sigma_shift_video", DEFAULT_SIGMA_SHIFT_VIDEO)) + shift_a = float(opts.get("minimax_h3_sigma_shift_audio", DEFAULT_SIGMA_SHIFT_AUDIO)) + sigma_v = (timestep.reshape(-1)[0] / 1000.0).to(torch.float32).clamp(min=1e-6) + sigma_a = time_shift_sigma(sigma_v, shift_v, shift_a) + t_video = float(1.0 - sigma_v) + t_audio = float(1.0 - sigma_a) + t_pin_v = max(t_video, MINIMAX_H3_IMGVID_COND_TIMESTEP) + t_pin_a = max(t_audio, MINIMAX_H3_AUDIO_REF_COND_TIMESTEP) + # ComfyUI only remaps the video/audio *target* segments. Text / pad keep t_video. + video_rows_t = mask_row_values(denoise_mask, latent_t, latent_h, latent_w) + if video_rows_t is not None: + video_rows_t = (1.0 - video_rows_t * float(sigma_v)).clamp(max=t_pin_v) + audio_rows_t = _audio_mask_values(audio_denoise_mask) + if audio_rows_t is not None: + audio_rows_t = (1.0 - audio_rows_t * float(sigma_a)).clamp(max=t_pin_a) + + return { + "packed": packed, + "text_embeddings": text, + "token_tags": _token_tags_for_payload(packed, payload), + "video_rows": video_rows, + "audio_rows": audio_rows, + "t_video": t_video, + "t_audio": t_audio, + "video_rows_t": video_rows_t, + "audio_rows_t": audio_rows_t, + "used": used, + "orig_video_shape": orig_shape, + "padded_video_shape": tuple(video.shape), + "sample_sigmas": sample_sigmas, + "sigma_v": float(sigma_v), + "shift_v": shift_v, + "shift_a": shift_a, + "imgvid_cond_noise_aug": float( + payload.get("visual_cond_noise_aug", MINIMAX_H3_IMGVID_COND_TIMESTEP) + ), + "audio_ref_cond_noise_aug": float( + payload.get("audio_cond_noise_aug", MINIMAX_H3_AUDIO_REF_COND_TIMESTEP) + ), + } + + +def build_step_forward_kwargs( + inputs: dict[str, Any], + *, + branch: MiniMaxH3DenoiseBranch | None = None, + device: torch.device | None = None, +) -> tuple[dict[str, Any], MiniMaxH3DenoiseBranch]: + """Assemble DiT ``forward`` kwargs from ``comfyui_payload_to_branch_inputs``.""" + if device is None: + device = inputs["video_rows"].device + if branch is None: + branch = MiniMaxH3DenoiseBranch( + packed=inputs["packed"], + text_embeddings=inputs["text_embeddings"], + token_tags=inputs["token_tags"], + device=device, + ) + plan = branch.prepare_timestep_plan( + video_timesteps=[inputs["t_video"]], + audio_timesteps=[inputs["t_audio"]], + imgvid_cond_noise_aug=inputs["imgvid_cond_noise_aug"], + audio_ref_cond_noise_aug=inputs["audio_ref_cond_noise_aug"], + ) + fk = branch.forward_kwargs( + video_rows=inputs["video_rows"].to(device=device, dtype=torch.float32), + audio_rows=inputs["audio_rows"].to(device=device, dtype=torch.float32), + step_timesteps=plan[0], + ) + fk = _overlay_per_row_timesteps( + fk, branch, inputs.get("video_rows_t"), inputs.get("audio_rows_t") + ) + extra = set(fk) - _FORWARD_SUPPORTED_KWARGS + if extra: + raise TypeError( + f"MiniMaxH3 ComfyUI step produced unsupported forward kwargs: {sorted(extra)}" + ) + return fk, branch + + +def pack_comfy_output( + v_video: torch.Tensor, + v_audio: torch.Tensor, + state: MiniMaxH3ComfyUIRunState, +) -> list[torch.Tensor]: + _b, _c, orig_t, orig_h, orig_w = state.orig_video_shape + _pb, _pc, padded_t, padded_h, padded_w = state.padded_video_shape + video = unpatchify_video( + v_video, + padded_t, + padded_h // 2, + padded_w // 2, + c=int(state.orig_video_shape[1]), + patch_size=_PATCH_SIZE, + ) + video = video[:, :, :orig_t, :orig_h, :orig_w] + # DiT returns target video rows, but audio_pos includes reference rows. + audio = unpack_audio(v_audio[state.branch.audio_target_slice]) + return [video, audio] + + +def _sample_sigmas_len(sample_sigmas: Any) -> int | None: + if sample_sigmas is None: + return None + if torch.is_tensor(sample_sigmas): + return int(sample_sigmas.reshape(-1).numel()) + if isinstance(sample_sigmas, (list, tuple)): + return len(sample_sigmas) + return None + + +@dataclass +class MiniMaxH3ComfyUIRunState: + branch: MiniMaxH3DenoiseBranch + used: int + orig_video_shape: tuple[int, ...] + padded_video_shape: tuple[int, ...] + sample_sigmas: Any = None + cache_mounted: bool = False + refined: bool = False + steps_done: int = 0 + signature: tuple[Any, ...] | None = None + + +class MiniMaxH3ComfyUIStepStage(PipelineStage): + def __init__(self, transformer) -> None: + super().__init__() + self.transformer = transformer + self._cache_stage = None + + def verify_input(self, batch: Req, server_args: ServerArgs): + bind_comfyui_session(batch) + return super().verify_input(batch, server_args) + + def _cache_dit_stage(self): + if self._cache_stage is None: + from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.denoising import ( + MiniMaxH3DenoisingStage, + ) + + self._cache_stage = MiniMaxH3DenoisingStage(transformer=self.transformer) + return self._cache_stage + + def _maybe_mount_cache_dit( + self, batch: Req, state: MiniMaxH3ComfyUIRunState + ) -> None: + if state.cache_mounted: + return + n_sigmas = _sample_sigmas_len(state.sample_sigmas) + if n_sigmas is None or n_sigmas < 2: + return + self._cache_dit_stage()._maybe_enable_cache_dit(n_sigmas - 1, batch) + state.cache_mounted = True + + def _release_run_state(self, state: MiniMaxH3ComfyUIRunState | None) -> None: + if state is None: + return + if state.cache_mounted and self._cache_stage is not None: + self._cache_stage._unmount_cache_dit() + state.cache_mounted = False + state.branch = None + + def _read_step_tensors( + self, batch: Req + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + extra = getattr(batch, "extra", None) or {} + video = batch.latents + audio = batch.audio_latents + if audio is None: + audio = extra.get("audio_latents") + context = extra.get("h3_context") + if context is None: + embeds = batch.prompt_embeds + if isinstance(embeds, list) and embeds: + context = embeds[0] + else: + context = embeds + if video is None or audio is None or context is None: + raise ValueError( + "MiniMaxH3 ComfyUI step requires latents, audio_latents, and prompt_embeds" + ) + return video, audio, context + + def _bind_run_state( + self, + batch: Req, + inputs: dict[str, Any], + sample_sigmas: Any, + device: torch.device, + ) -> MiniMaxH3ComfyUIRunState: + from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.denoising import ( + _precompute_refined_prompt_embeds, + ) + + signature = ( + inputs["orig_video_shape"], + inputs["used"], + int(inputs["text_embeddings"].shape[0]), + ) + existing = get_run_state(batch) + sigmas = _sigma_list(sample_sigmas) + is_first_sigma = False + if sigmas: + is_first_sigma = abs(float(inputs["sigma_v"]) - sigmas[0]) < 1e-4 + if existing is not None and ( + existing.signature != signature + or (is_first_sigma and existing.steps_done > 0) + ): + self._release_run_state(existing) + pop_run_state(batch) + existing = None + + def _factory() -> MiniMaxH3ComfyUIRunState: + branch = MiniMaxH3DenoiseBranch( + packed=inputs["packed"], + text_embeddings=inputs["text_embeddings"], + token_tags=inputs["token_tags"], + device=device, + ) + return MiniMaxH3ComfyUIRunState( + branch=branch, + used=inputs["used"], + orig_video_shape=inputs["orig_video_shape"], + padded_video_shape=inputs["padded_video_shape"], + sample_sigmas=sample_sigmas, + signature=signature, + ) + + state = ( + existing + if existing is not None + else get_or_create_run_state(batch, _factory) + ) + if not state.refined: + _precompute_refined_prompt_embeds( + self.transformer, state.branch, device=device + ) + state.refined = True + return state + + def forward(self, batch: Req, server_args: ServerArgs) -> Req: + bind_comfyui_session(batch) + extra = getattr(batch, "extra", None) or {} + video_x, audio_x, context = self._read_step_tensors(batch) + payload = extra.get("h3_payload") or {} + transformer_options = extra.get("h3_transformer_options") or {} + sample_sigmas = extra.get("h3_sample_sigmas") + if sample_sigmas is None: + sample_sigmas = batch.sigmas + timestep = batch.timesteps + if timestep is None: + timestep = batch.timestep + if timestep is None: + raise ValueError("MiniMaxH3 ComfyUI step requires timesteps") + + inputs = comfyui_payload_to_branch_inputs( + video_x, + audio_x, + context, + payload, + sample_sigmas, + timestep, + transformer_options=transformer_options, + layout=extra.get("h3_layout"), + denoise_mask=extra.get("h3_denoise_mask"), + audio_denoise_mask=extra.get("h3_audio_denoise_mask"), + ) + state = self._bind_run_state(batch, inputs, sample_sigmas, video_x.device) + self._maybe_mount_cache_dit(batch, state) + fk, _branch = build_step_forward_kwargs( + inputs, branch=state.branch, device=video_x.device + ) + v_video, v_audio = self.transformer(**fk) + batch.noise_pred = pack_comfy_output(v_video, v_audio, state) + state.steps_done += 1 + n_sigmas = _sample_sigmas_len(sample_sigmas) + if n_sigmas is not None and n_sigmas >= 2 and state.steps_done >= n_sigmas - 1: + self._release_run_state(state) + release_run_state(session_id_from_req(batch)) + return batch + + +__all__ = [ + "DEFAULT_SIGMA_SHIFT_AUDIO", + "DEFAULT_SIGMA_SHIFT_VIDEO", + "MiniMaxH3ComfyUIRunState", + "MiniMaxH3ComfyUIStepStage", + "build_step_forward_kwargs", + "comfyui_layout_to_packed", + "comfyui_payload_to_branch_inputs", + "mask_row_values", + "pack_audio", + "pack_comfy_output", + "patchify_video", + "serialize_comfyui_layout", + "time_shift_sigma", + "time_shift_slope", + "unpack_audio", + "unpatchify_video", +] diff --git a/python/sglang/multimodal_gen/runtime/scheduler_client.py b/python/sglang/multimodal_gen/runtime/scheduler_client.py index 27054bc4e1..df6ab5a205 100644 --- a/python/sglang/multimodal_gen/runtime/scheduler_client.py +++ b/python/sglang/multimodal_gen/runtime/scheduler_client.py @@ -7,6 +7,11 @@ from typing import Any, Optional import zmq import zmq.asyncio +from sglang.multimodal_gen.runtime.distributed.ipc_cuda import ( + materialize_cuda_refs, + release_retained_producer_tensors, + spill_cuda_tensors, +) from sglang.multimodal_gen.runtime.entrypoints.control_requests import ( ListLorasReq, MergeLoraWeightsReq, @@ -22,7 +27,10 @@ from sglang.multimodal_gen.runtime.entrypoints.post_training.io_struct import ( UpdateWeightFromTensorCheckerReqInput, UpdateWeightFromTensorReqInput, ) -from sglang.multimodal_gen.runtime.ipc_array import materialize_file_refs +from sglang.multimodal_gen.runtime.ipc_array import ( + is_local_endpoint, + materialize_file_refs, +) from sglang.multimodal_gen.runtime.pipelines_core import Req from sglang.multimodal_gen.runtime.pipelines_core.schedule_batch import OutputBatch from sglang.multimodal_gen.runtime.server_args import ( @@ -174,9 +182,10 @@ class SchedulerClient: effective_timeout = _resolve_timeout_ms(self.server_args, timeout_ms) _configure_recv_timeout(socket, effective_timeout) socket.connect(endpoint) - socket.send_pyobj(batch) + socket.send_pyobj(_prepare_local_payload(endpoint, batch)) output_batch = socket.recv_pyobj() _materialize_output_batch_file_refs(output_batch) + _materialize_local_cuda_refs(endpoint, output_batch) return output_batch except zmq.error.Again: logger.error("Timeout waiting for response from %s.", endpoint) @@ -286,10 +295,11 @@ class AsyncSchedulerClient: effective_timeout = _resolve_timeout_ms(self.server_args, timeout_ms) _configure_recv_timeout(socket, effective_timeout) socket.connect(endpoint) - await socket.send(pickle.dumps(batch)) + await socket.send(pickle.dumps(_prepare_local_payload(endpoint, batch))) payload = await socket.recv() output_batch = pickle.loads(payload) _materialize_output_batch_file_refs(output_batch) + _materialize_local_cuda_refs(endpoint, output_batch) return output_batch except zmq.error.Again: logger.error("Timeout waiting for response from %s.", endpoint) @@ -331,6 +341,19 @@ async_scheduler_client = AsyncSchedulerClient() sync_scheduler_client = SchedulerClient() +def _prepare_local_payload(endpoint: str, batch: Any) -> Any: + if is_local_endpoint(endpoint): + return spill_cuda_tensors(batch) + return batch + + +def _materialize_local_cuda_refs(endpoint: str, output_batch: Any) -> None: + if is_local_endpoint(endpoint): + materialize_cuda_refs(output_batch) + # Request-side IPC retains are no longer needed after the round-trip. + release_retained_producer_tensors() + + def _materialize_output_batch_file_refs(output_batch: Any) -> None: if not isinstance(output_batch, OutputBatch): return diff --git a/python/sglang/multimodal_gen/test/unit/test_comfyui_adapters.py b/python/sglang/multimodal_gen/test/unit/test_comfyui_adapters.py new file mode 100644 index 0000000000..42c1772baf --- /dev/null +++ b/python/sglang/multimodal_gen/test/unit/test_comfyui_adapters.py @@ -0,0 +1,58 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Pack / unpack contract for ComfyUI model adapters.""" + +import torch + +from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.adapter import ( + get_adapter_class, + registered_model_types, +) +from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.flux import FluxAdapter +from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.zimage import ( + ZImageAdapter, +) + + +def test_registered_comfyui_model_types() -> None: + types = registered_model_types() + assert "flux" in types + assert "lumina2" in types + assert get_adapter_class("lumina2") is ZImageAdapter + assert get_adapter_class("flux") is FluxAdapter + assert get_adapter_class("lumina2").pipeline_class_name == "ZImagePipeline" + + +def test_zimage_pack_sets_seq_lens_and_time_dim() -> None: + adapter = ZImageAdapter() + x = torch.ones(1, 16, 90, 160) + timestep = torch.tensor([1.0]) + context = torch.ones(1, 19, 2560) + packed = adapter.pack(x, timestep, context) + assert packed.latents.shape == (1, 16, 1, 90, 160) + assert packed.prompt_embeds[0].shape == (19, 2560) + assert packed.prompt_seq_lens == [[19]] + assert packed.height == 720 + assert packed.width == 1280 + assert torch.equal(packed.timesteps, timestep * 1000.0) + + pred = torch.ones(1, 16, 1, 90, 160) + out = adapter.unpack(pred, packed, x) + assert out.shape == x.shape + + +def test_flux_pack_and_unpack_roundtrip() -> None: + adapter = FluxAdapter() + x = torch.arange(1 * 16 * 8 * 8, dtype=torch.float32).reshape(1, 16, 8, 8) + timestep = torch.tensor([0.5]) + context = torch.ones(1, 8, 4096) + y = torch.ones(1, 768) + packed = adapter.pack(x, timestep, context, y=y, guidance=torch.tensor([1.0])) + assert packed.latents.ndim == 3 + assert packed.pooled_embeds[0] is y + assert packed.guidance_scale == 1.0 + out = adapter.unpack(packed.latents, packed, x) + assert out.shape == x.shape + assert torch.equal(out, x) + + default = adapter.pack(x, timestep, context, y=y) + assert default.guidance_scale == 3.5 diff --git a/python/sglang/multimodal_gen/test/unit/test_comfyui_generator.py b/python/sglang/multimodal_gen/test/unit/test_comfyui_generator.py new file mode 100644 index 0000000000..d89d7206a1 --- /dev/null +++ b/python/sglang/multimodal_gen/test/unit/test_comfyui_generator.py @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Process-wide SGLD worker ownership for ComfyUI loaders.""" + +from types import SimpleNamespace + +from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.core.generator import ( + SGLDiffusionGenerator, +) + + +def test_shared_is_process_singleton() -> None: + SGLDiffusionGenerator.reset_shared() + assert SGLDiffusionGenerator.shared() is SGLDiffusionGenerator.shared() + SGLDiffusionGenerator.reset_shared() + + +def test_reuse_requires_live_worker() -> None: + runtime = SGLDiffusionGenerator() + options = {"model_path": "z.safetensors"} + runtime.last_options = options + runtime.generator = object() + runtime._patcher = object() + runtime.executor = object() + runtime._is_live = lambda: False + assert runtime._can_reuse(options) is False + runtime._is_live = lambda: True + assert runtime._can_reuse(options) is True + assert runtime._can_reuse({"model_path": "h3.safetensors"}) is False + + +def test_ensure_rebuilds_when_another_model_owns_the_worker() -> None: + runtime = SGLDiffusionGenerator() + stale = object() + fresh = object() + loads = [] + executor = SimpleNamespace( + generator=stale, + _sgld_reload={ + "model_path": "z.safetensors", + "model_options": {}, + "sgld_options": {}, + }, + _lora_input=None, + ) + + def fake_load(**kwargs): + loads.append(kwargs) + runtime.generator = fresh + return "patcher" + + runtime.load_model = fake_load + runtime.ensure_executor(executor) + assert loads == [executor._sgld_reload] + assert executor.generator is fresh + + +def test_ensure_is_noop_when_executor_still_owns_live_worker() -> None: + runtime = SGLDiffusionGenerator() + gen = object() + runtime.generator = gen + runtime._is_live = lambda: True + executor = SimpleNamespace(generator=gen, _sgld_reload={"model_path": "z"}) + runtime.load_model = lambda **kwargs: (_ for _ in ()).throw( + AssertionError("should not reload") + ) + runtime.ensure_executor(executor) + assert executor.generator is gen + + +def test_kill_generator_only_touches_owned_workers() -> None: + runtime = SGLDiffusionGenerator() + owned = SimpleNamespace(alive=True, terminated=False, killed=False, pid=9) + + def terminate(): + owned.terminated = True + owned.alive = False + + owned.is_alive = lambda: owned.alive + owned.terminate = terminate + owned.join = lambda timeout=None: None + owned.kill = lambda: setattr(owned, "killed", True) + runtime.generator = SimpleNamespace(local_scheduler_process=[owned]) + runtime.kill_generator() + assert owned.terminated is True + assert owned.killed is False diff --git a/python/sglang/multimodal_gen/test/unit/test_comfyui_h3.py b/python/sglang/multimodal_gen/test/unit/test_comfyui_h3.py new file mode 100644 index 0000000000..884b300e5f --- /dev/null +++ b/python/sglang/multimodal_gen/test/unit/test_comfyui_h3.py @@ -0,0 +1,1100 @@ +# SPDX-License-Identifier: Apache-2.0 +"""MiniMax-H3 ComfyUI contracts: layout, adapter, step kwargs, profile, QKV.""" + +from __future__ import annotations + +import math + +import pytest +import torch + +from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.packed_sequence import ( + minimax_h3_packed_sequence, + minimax_h3_packed_sequence_ref2va_blocks, +) +from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import ( + comfyui_layout_to_packed, + serialize_comfyui_layout, +) + +FRAME_PER_TOKEN = (1, 4, 4, 4, 4) +FRAME_RESCALE = 5.0 / 3.0 + + +def _axis_from_sqrt_area(dim, patch, sqrt_area): + ratio = dim / sqrt_area + n = dim // patch + return ( + torch.arange(n, dtype=torch.float64) * (ratio / n) + (1.0 - ratio) / 2.0 + ) * 32.0 + + +def _frame_grid(h, w): + area = math.sqrt(h * w) + hh, ww = torch.meshgrid( + _axis_from_sqrt_area(h, 2, area), + _axis_from_sqrt_area(w, 2, area), + indexing="ij", + ) + return torch.stack([hh.reshape(-1), ww.reshape(-1)], dim=-1), _axis_from_sqrt_area( + w, 2, area + ) + + +def _video_t_spans(n): + return [FRAME_RESCALE * FRAME_PER_TOKEN[k % 5] for k in range(n)] + + +def _video_t_grid(n, origin): + spans = torch.tensor(_video_t_spans(n), dtype=torch.float64) + return float(origin) + torch.cat( + [torch.zeros(1, dtype=torch.float64), spans[:-1].cumsum(0)] + ) + + +def _ref_t_span(blk): + kind = blk["kind"] + if kind == "image": + return 1.0 + if kind == "audio": + return float(blk["ref_audio_t"]) + if kind in ("video", "video_audio"): + return max(float(blk["ref_audio_t"]), sum(_video_t_spans(blk["latent_t"]))) + return 0.0 + + +def _audio_grid(cursor, t, w_low, w_high): + g = torch.zeros(t * 2, 3, dtype=torch.float64) + g[:, 0] = (cursor + torch.arange(t, dtype=torch.float64)).repeat(2) + g[:t, 2] = w_low + g[t:, 2] = w_high + return g + + +def _video_grid(vt, frame, cursor): + g = torch.empty(vt, frame.shape[0], 3, dtype=torch.float64) + g[:, :, 0] = _video_t_grid(vt, cursor)[:, None] + g[:, :, 1:] = frame[None] + return g.reshape(-1, 3) + + +class ComfyUIPackedLayout: + """Standalone replica of ComfyUI ``PackedLayout`` (no comfy import).""" + + def __init__( + self, text_len, latent_t, latent_h, latent_w, audio_t, keyframes=None, refs=None + ): + frame, w_grid = _frame_grid(latent_h, latent_w) + frame_rows = frame.shape[0] + + segments = [("text", text_len)] + g = torch.zeros(text_len, 3, dtype=torch.float64) + g[:, 0] = torch.arange(text_len, dtype=torch.float64) + pos = [g] + + img_pos, img_update = [], [] + audio_pos, audio_update = [], [] + row = text_len + + target_audio_w = (float(w_grid[0]), float(w_grid[-1])) + cursor = float(text_len) + for blk in refs or (): + cursor += _ref_t_span(blk) + + if keyframes: + for kf in keyframes: + cond_t = cursor + FRAME_RESCALE * kf["resolved_frame_index"] + video_latent = kf.get("latent") + if video_latent is not None: + vt = video_latent.shape[2] + n = vt * frame_rows + segments.append(("cond", n)) + pos.append(_video_grid(vt, frame, cond_t)) + img_pos.append(torch.arange(row, row + n)) + img_update.append(torch.zeros(n, dtype=torch.bool)) + row += n + audio_latent = kf.get("audio_latent") + if audio_latent is not None: + rt = audio_latent.shape[-1] + segments.append(("cond_audio", rt * 2)) + pos.append(_audio_grid(cond_t, rt, *target_audio_w)) + audio_pos.append(torch.arange(row, row + rt * 2)) + audio_update.append(torch.zeros(rt * 2, dtype=torch.bool)) + row += rt * 2 + + if refs: + cursor = float(text_len) + for blk in refs: + kind = blk["kind"] + if kind == "image": + r_frame, _ = _frame_grid(blk["latent_h"], blk["latent_w"]) + n = r_frame.shape[0] + g = torch.empty(n, 3, dtype=torch.float64) + g[:, 0] = cursor + g[:, 1:] = r_frame + segments.append(("ref_img", n)) + pos.append(g) + img_pos.append(torch.arange(row, row + n)) + img_update.append(torch.zeros(n, dtype=torch.bool)) + row += n + cursor += 1.0 + elif kind == "audio": + rt = blk["ref_audio_t"] + if rt > 0: + segments.append(("ref_audio", rt * 2)) + pos.append(_audio_grid(cursor, rt, *target_audio_w)) + audio_pos.append(torch.arange(row, row + rt * 2)) + audio_update.append(torch.zeros(rt * 2, dtype=torch.bool)) + row += rt * 2 + cursor += float(rt) + elif kind in ("video", "video_audio"): + rt = blk["ref_audio_t"] + vt = blk["latent_t"] + r_frame, r_w_grid = _frame_grid(blk["latent_h"], blk["latent_w"]) + if rt > 0: + segments.append(("ref_audio", rt * 2)) + pos.append( + _audio_grid( + cursor, rt, float(r_w_grid[0]), float(r_w_grid[-1]) + ) + ) + audio_pos.append(torch.arange(row, row + rt * 2)) + audio_update.append(torch.zeros(rt * 2, dtype=torch.bool)) + row += rt * 2 + n = vt * r_frame.shape[0] + segments.append(("ref_img", n)) + pos.append(_video_grid(vt, r_frame, cursor)) + img_pos.append(torch.arange(row, row + n)) + img_update.append(torch.zeros(n, dtype=torch.bool)) + row += n + cursor += max(float(rt), sum(_video_t_spans(vt))) + + segments.append(("audio", audio_t * 2)) + pos.append(_audio_grid(cursor, audio_t, *target_audio_w)) + audio_pos.append(torch.arange(row, row + audio_t * 2)) + audio_update.append(torch.ones(audio_t * 2, dtype=torch.bool)) + row += audio_t * 2 + + n_video = latent_t * frame_rows + segments.append(("video", n_video)) + pos.append(_video_grid(latent_t, frame, cursor)) + img_pos.append(torch.arange(row, row + n_video)) + img_update.append(torch.ones(n_video, dtype=torch.bool)) + row += n_video + + self.seq_len = row + self.position_ids = torch.cat(pos) + self.img_pos = torch.cat(img_pos) + self.img_update = torch.cat(img_update) + self.audio_pos = torch.cat(audio_pos) + self.audio_update = torch.cat(audio_update) + self.signature = (text_len, latent_t, latent_h, latent_w, audio_t) + seg_abs = [] + off = 0 + for kind, n in segments: + seg_abs.append((off, off + n, kind)) + off += n + self.segments = seg_abs + + +def _sgld_used(packed) -> int: + return int(packed["cu_seqlens"][1]) + + +def _sgld_segment_kinds(packed, text_len: int) -> list[str]: + used = _sgld_used(packed) + img_set = {int(i) for i in packed["img_pos"].tolist()} + audio_set = {int(i) for i in packed["audio_pos"].tolist()} + update = { + int(pos): bool(flag) + for pos, flag in zip(packed["img_pos"].tolist(), packed["update_mask"].tolist()) + } + audio_update_mask = packed.get("audio_update_mask") + if audio_update_mask is None: + audio_update = {int(i): True for i in packed["audio_pos"].tolist()} + else: + audio_update = { + int(pos): bool(flag) + for pos, flag in zip( + packed["audio_pos"].tolist(), audio_update_mask.tolist() + ) + } + + kinds: list[str] = [] + for row in range(used): + if row < text_len: + kind = "text" + elif row in img_set: + kind = "video" if update[row] else "cond_or_ref_img" + elif row in audio_set: + kind = "audio" if audio_update[row] else "cond_or_ref_audio" + else: + kind = "unknown" + if not kinds or kinds[-1] != kind: + kinds.append(kind) + return kinds + + +def _comfy_coarse_kinds(layout: ComfyUIPackedLayout) -> list[str]: + mapping = { + "text": "text", + "cond": "cond_or_ref_img", + "ref_img": "cond_or_ref_img", + "cond_audio": "cond_or_ref_audio", + "ref_audio": "cond_or_ref_audio", + "audio": "audio", + "video": "video", + } + kinds: list[str] = [] + for _, _, kind in layout.segments: + mapped = mapping[kind] + if not kinds or kinds[-1] != mapped: + kinds.append(mapped) + return kinds + + +def _assert_used_prefix_aligned(comfy: ComfyUIPackedLayout, packed: dict) -> None: + used = _sgld_used(packed) + assert used == comfy.seq_len + assert _sgld_segment_kinds(packed, comfy.signature[0]) == _comfy_coarse_kinds(comfy) + assert torch.equal(packed["img_pos"], comfy.img_pos) + assert torch.equal(packed["audio_pos"], comfy.audio_pos) + assert torch.equal(packed["update_mask"], comfy.img_update) + sgld_audio_update = packed.get("audio_update_mask") + if sgld_audio_update is None: + assert bool(comfy.audio_update.all()) + else: + assert torch.equal(sgld_audio_update, comfy.audio_update) + sgld_pos = packed["img_position_ids"][:used].to(torch.float32) + comfy_pos = comfy.position_ids.to(torch.float32) + assert torch.equal(sgld_pos, comfy_pos) + + +def test_t2va_used_prefix_matches_comfyui(): + kwargs = dict(text_len=8, latent_t=2, latent_h=4, latent_w=4, audio_t=3) + comfy = ComfyUIPackedLayout(**kwargs) + packed = minimax_h3_packed_sequence(**kwargs, include_keyframe_cond=False) + _assert_used_prefix_aligned(comfy, packed) + assert [k for _, _, k in comfy.segments] == ["text", "audio", "video"] + + +def test_fl2va_first_last_used_prefix_matches_comfyui(): + text_len, latent_t, latent_h, latent_w, audio_t = 11, 37, 48, 76, 203 + frame_count = 124 + comfy = ComfyUIPackedLayout( + text_len, + latent_t, + latent_h, + latent_w, + audio_t, + keyframes=[ + { + "resolved_frame_index": 0, + "latent": torch.zeros(1, 24, 1, latent_h, latent_w), + }, + { + "resolved_frame_index": frame_count - 1, + "latent": torch.zeros(1, 24, 1, latent_h, latent_w), + }, + ], + ) + packed = minimax_h3_packed_sequence( + text_len=text_len, + latent_t=latent_t, + latent_h=latent_h, + latent_w=latent_w, + audio_t=audio_t, + include_keyframe_cond=True, + keyframe_frame_indices=[0, -1], + frame_count=frame_count, + ) + _assert_used_prefix_aligned(comfy, packed) + assert [k for _, _, k in comfy.segments] == [ + "text", + "cond", + "cond", + "audio", + "video", + ] + + +def test_ref2va_image_plus_video_audio_used_prefix_matches_comfyui(): + kwargs = dict(text_len=5, latent_t=2, latent_h=4, latent_w=4, audio_t=5) + refs = [ + {"kind": "image", "latent_h": 4, "latent_w": 4}, + { + "kind": "video_audio", + "ref_audio_t": 3, + "latent_t": 2, + "latent_h": 4, + "latent_w": 4, + }, + ] + comfy = ComfyUIPackedLayout(**kwargs, refs=refs) + packed = minimax_h3_packed_sequence_ref2va_blocks(**kwargs, ref_blocks=refs) + _assert_used_prefix_aligned(comfy, packed) + assert [k for _, _, k in comfy.segments] == [ + "text", + "ref_img", + "ref_audio", + "ref_img", + "audio", + "video", + ] + + +@pytest.mark.xfail(strict=True, reason="D3: SGLD packed_sequence has no cond_audio") +def test_gap_keyframe_audio_latent_cond_audio(): + latent_h, latent_w = 4, 4 + comfy = ComfyUIPackedLayout( + 4, + 2, + latent_h, + latent_w, + 3, + keyframes=[ + { + "resolved_frame_index": 0, + "latent": torch.zeros(1, 24, 1, latent_h, latent_w), + "audio_latent": torch.zeros(1, 32, 2, 2), + } + ], + ) + assert any(kind == "cond_audio" for _, _, kind in comfy.segments) + packed = minimax_h3_packed_sequence( + text_len=4, + latent_t=2, + latent_h=latent_h, + latent_w=latent_w, + audio_t=3, + include_keyframe_cond=True, + keyframe_frame_indices=[0], + frame_count=5, + ) + comfy_cond_audio = sum( + stop - start for start, stop, kind in comfy.segments if kind == "cond_audio" + ) + sgld_audio_cond = int((~packed.get("audio_update_mask", torch.ones(1))).sum()) + assert sgld_audio_cond == comfy_cond_audio + + +@pytest.mark.xfail( + strict=True, reason="D3: SGLD only accepts keyframe signatures (0,), (-1,), (0, -1)" +) +def test_gap_arbitrary_frame_idx(): + packed = minimax_h3_packed_sequence( + text_len=4, + latent_t=37, + latent_h=4, + latent_w=4, + audio_t=3, + include_keyframe_cond=True, + keyframe_frame_indices=[51], + frame_count=124, + ) + comfy = ComfyUIPackedLayout( + 4, + 37, + 4, + 4, + 3, + keyframes=[ + {"resolved_frame_index": 51, "latent": torch.zeros(1, 24, 1, 4, 4)}, + ], + ) + _assert_used_prefix_aligned(comfy, packed) + + +def test_translated_layout_covers_keyframe_audio_and_arbitrary_idx(): + latent_h, latent_w = 4, 4 + comfy = ComfyUIPackedLayout( + 4, + 2, + latent_h, + latent_w, + 3, + keyframes=[ + { + "resolved_frame_index": 51, + "latent": torch.zeros(1, 24, 1, latent_h, latent_w), + "audio_latent": torch.zeros(1, 32, 2, 2), + } + ], + ) + packed = comfyui_layout_to_packed(serialize_comfyui_layout(comfy)) + _assert_used_prefix_aligned(comfy, packed) + assert packed["seq_len"] % 64 == 0 + assert packed["seq_len"] >= comfy.seq_len + assert any(kind == "cond_audio" for _, _, kind in comfy.segments) + assert int((~packed["audio_update_mask"]).sum()) > 0 + + +@pytest.mark.xfail( + strict=True, + reason="D3: ComfyUI does not pad; SGLD pads seq_len to a multiple of 64", +) +def test_gap_sgld_pads_seq_len_to_64(): + kwargs = dict(text_len=8, latent_t=2, latent_h=4, latent_w=4, audio_t=3) + comfy = ComfyUIPackedLayout(**kwargs) + packed = minimax_h3_packed_sequence(**kwargs, include_keyframe_cond=False) + assert packed["seq_len"] == comfy.seq_len + assert packed["seq_len"] % 64 == 0 + assert int((packed["token_tags"] < 0).sum()) > 0 + + +# --- adapter --- + +import torch + +from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.adapter import ( + get_adapter_class, + registered_model_types, +) +from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.base import ( + SGLDiffusionExecutor, +) +from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.minimax_h3 import ( + MiniMaxH3Adapter, + drop_h3_pinned_sampling_fields, + worker_transformer_options, +) +from sglang.multimodal_gen.configs.sample.minimax_h3 import MiniMaxH3SamplingParams + + +def test_begin_sampler_run_bumps_id_and_clears_sent_conds() -> None: + class _State: + session_id = "abc" + _run_id = 0 + _sent_conds = {("old",)} + + begin_sampler_run = SGLDiffusionExecutor.begin_sampler_run + comfyui_session_id = SGLDiffusionExecutor.comfyui_session_id + + state = _State() + SGLDiffusionExecutor.begin_sampler_run(state) + assert state._run_id == 1 + assert state._sent_conds == set() + assert SGLDiffusionExecutor.comfyui_session_id(state) == "abc:1" + SGLDiffusionExecutor.begin_sampler_run(state) + assert state._run_id == 2 + assert SGLDiffusionExecutor.comfyui_session_id(state) == "abc:2" + + +def test_h3_sampling_params_kwargs_omit_pinned_fields() -> None: + kwargs = drop_h3_pinned_sampling_fields( + { + "prompt": " ", + "guidance_scale": 4.5, + "height": 48, + "width": 28, + "num_frames": 1, + "num_inference_steps": 1, + "save_output": False, + "suppress_logs": True, + } + ) + MiniMaxH3SamplingParams(**kwargs) + assert "guidance_scale" not in kwargs + assert "num_frames" not in kwargs + + +def test_worker_transformer_options_drop_comfy_objects() -> None: + class ModelSampling: + pass + + opts = worker_transformer_options( + { + "minimax_h3_sigma_shift_video": 12.0, + "sample_sigmas": torch.tensor([1.0, 0.0]), + "model_sampling": ModelSampling(), + "patches": {"x": 1}, + } + ) + assert opts["minimax_h3_sigma_shift_video"] == 12.0 + assert "model_sampling" not in opts + assert "patches" not in opts + + +def test_minimax_h3_is_registered() -> None: + types = registered_model_types() + assert "minimax_h3" in types + assert get_adapter_class("minimax_h3") is MiniMaxH3Adapter + assert get_adapter_class("minimax_h3").pipeline_class_name == "MiniMaxH3Pipeline" + + +def test_h3_pack_fills_extra_req_and_unpack_restores_shapes() -> None: + adapter = MiniMaxH3Adapter() + video = torch.ones(1, 24, 2, 4, 4) + audio = torch.ones(1, 32, 2, 3) * 2 + x = [video, audio] + timestep = torch.tensor([500.0]) + context = torch.ones(1, 8, 16) + payload = {"audio_scale": 1.0, "layout": object()} + sample_sigmas = torch.tensor([1.0, 0.5, 0.0]) + packed = adapter.pack( + x, + timestep, + context, + minimax_payload=payload, + transformer_options={"sample_sigmas": sample_sigmas}, + ) + assert packed.latents.shape == video.shape + assert packed.extra_req["audio_latents"].shape == audio.shape + assert packed.extra_req["h3_payload"]["audio_scale"] == 1.0 + assert "layout" not in packed.extra_req["h3_payload"] + assert "h3_layout" not in packed.extra_req + assert torch.equal(packed.extra_req["h3_sample_sigmas"], sample_sigmas) + assert packed.extra_req["comfyui_cache_fp"]["spatial"] == (2, 4, 4) + assert packed.extra_req["comfyui_cache_fp"]["sigmas"] == (1.0, 0.0, 3) + assert packed.prompt_embeds[0].shape == (8, 16) + assert packed.timesteps is timestep + + pred = [torch.zeros_like(video), torch.zeros_like(audio)] + out = adapter.unpack(pred, packed, x) + assert isinstance(out, list) + assert out[0].shape == video.shape + assert out[1].shape == audio.shape + + +def test_h3_audio_scale_is_undone_on_pack_and_restored_on_unpack() -> None: + adapter = MiniMaxH3Adapter() + video = torch.ones(1, 24, 2, 4, 4) + audio = torch.ones(1, 32, 2, 3) + x = [video, audio] + timestep = torch.tensor([200.0]) + context = torch.ones(1, 4, 8) + packed = adapter.pack( + x, + timestep, + context, + minimax_payload={"audio_scale": 0.5}, + transformer_options={ + "minimax_h3_sigma_shift_video": 12.0, + "minimax_h3_sigma_shift_audio": 3.0, + }, + ) + assert not torch.equal(packed.extra_req["audio_latents"], audio) + pred = [torch.ones_like(video), torch.ones_like(audio)] + out = adapter.unpack(pred, packed, x) + assert out[0].shape == video.shape + assert out[1].shape == audio.shape + + +# --- step kwargs --- + +import pytest +import torch + +from sglang.multimodal_gen.runtime.models.dits.minimax_h3 import ( + _FORWARD_SUPPORTED_KWARGS, +) +from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import ( + build_step_forward_kwargs, + comfyui_payload_to_branch_inputs, + pack_audio, + pack_comfy_output, + patchify_video, + serialize_comfyui_layout, + unpack_audio, + unpatchify_video, +) + + +def _t2va_tensors(): + video = torch.randn(1, 24, 2, 4, 4) + audio = torch.randn(1, 32, 2, 3) + context = torch.randn(1, 8, 16) + timestep = torch.tensor([500.0]) + sample_sigmas = torch.tensor([1.0, 0.5, 0.0]) + return video, audio, context, timestep, sample_sigmas + + +def test_t2va_forward_kwargs_are_supported_and_complete(): + video, audio, context, timestep, sample_sigmas = _t2va_tensors() + inputs = comfyui_payload_to_branch_inputs( + video, + audio, + context, + minimax_payload={}, + sample_sigmas=sample_sigmas, + timestep=timestep, + ) + assert inputs["used"] == 8 + 3 * 2 + 2 * 2 * 2 + assert inputs["packed"]["seq_len"] % 64 == 0 + assert inputs["packed"]["seq_len"] >= inputs["used"] + assert torch.equal(inputs["text_embeddings"], context[0]) + assert inputs["video_rows"].shape[0] == 2 * 2 * 2 + assert inputs["audio_rows"].shape[0] == 6 + + fk, branch = build_step_forward_kwargs(inputs, device=torch.device("cpu")) + assert set(fk) <= _FORWARD_SUPPORTED_KWARGS + for required in ( + "x", + "audio_x", + "img_position_ids", + "unique_timesteps", + "inverse_indices", + "update_mask", + "block_token_tags", + "skip_mask_out_condition", + "prompt_embeds", + "img_pos_info", + "audio_pos_info", + "text_pos_info", + "img_pos_for_infer_output_info", + "local_embedding_layout", + "packed_seq_params", + "refiner_packed_seq_params", + "block_combined_indices", + ): + assert required in fk, required + assert torch.equal(fk["prompt_embeds"], context[0].to(fk["prompt_embeds"].device)) + assert fk["x"].shape[1] == inputs["packed"]["seq_len"] + assert branch.seq_len == inputs["packed"]["seq_len"] + + +def test_context_is_used_as_refined_prompt_embeds(): + video, audio, _context, timestep, sample_sigmas = _t2va_tensors() + refined = torch.randn(8, 32) + inputs = comfyui_payload_to_branch_inputs( + video, + audio, + refined, + minimax_payload={}, + sample_sigmas=sample_sigmas, + timestep=timestep, + ) + fk, _branch = build_step_forward_kwargs(inputs, device=torch.device("cpu")) + assert torch.equal(fk["prompt_embeds"], refined) + + +def test_pack_comfy_output_crops_to_unpadded_shape(): + video, audio, context, timestep, sample_sigmas = _t2va_tensors() + inputs = comfyui_payload_to_branch_inputs( + video, + audio, + context, + minimax_payload={}, + sample_sigmas=sample_sigmas, + timestep=timestep, + ) + _fk, branch = build_step_forward_kwargs(inputs, device=torch.device("cpu")) + from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import ( + MiniMaxH3ComfyUIRunState, + ) + + state = MiniMaxH3ComfyUIRunState( + branch=branch, + used=inputs["used"], + orig_video_shape=inputs["orig_video_shape"], + padded_video_shape=inputs["padded_video_shape"], + ) + v_video = torch.zeros(inputs["video_rows"].shape) + v_audio = torch.zeros(inputs["audio_rows"].shape) + out_video, out_audio = pack_comfy_output(v_video, v_audio, state) + assert out_video.shape == video.shape + assert out_audio.shape == audio.shape + + +def test_patchify_roundtrip(): + video = torch.arange(1 * 24 * 2 * 4 * 4, dtype=torch.float32).reshape( + 1, 24, 2, 4, 4 + ) + rows = patchify_video(video) + restored = unpatchify_video(rows, 2, 2, 2) + assert torch.equal(restored, video) + audio = torch.arange(1 * 32 * 2 * 3, dtype=torch.float32).reshape(1, 32, 2, 3) + packed = pack_audio(audio) + assert torch.equal(unpack_audio(packed), audio) + + +def test_keyframes_without_layout_are_still_rejected(): + video, audio, context, timestep, sample_sigmas = _t2va_tensors() + with pytest.raises(NotImplementedError, match="PackedLayout"): + comfyui_payload_to_branch_inputs( + video, + audio, + context, + minimax_payload={"keyframes": [{"resolved_frame_index": 0}]}, + sample_sigmas=sample_sigmas, + timestep=timestep, + ) + + +def test_stale_layout_is_rebuilt_from_current_video_shape(): + video, audio, context, timestep, sample_sigmas = _t2va_tensors() + stale = ComfyUIPackedLayout(8, 2, 2, 2, 3) + inputs = comfyui_payload_to_branch_inputs( + video, + audio, + context, + minimax_payload={}, + sample_sigmas=sample_sigmas, + timestep=timestep, + layout=serialize_comfyui_layout(stale), + ) + assert stale.signature == (8, 2, 2, 2, 3) + assert tuple(video.shape[2:]) == (2, 4, 4) + assert inputs["video_rows"].shape[0] == 2 * 2 * 2 + assert inputs["used"] == 8 + 3 * 2 + 2 * 2 * 2 + + +def test_keyframes_with_serialized_layout_build_kwargs(): + video, audio, context, timestep, sample_sigmas = _t2va_tensors() + kf_latent = torch.randn(1, 24, 1, 4, 4) + layout = ComfyUIPackedLayout( + 8, + 2, + 4, + 4, + 3, + keyframes=[{"resolved_frame_index": 0, "latent": kf_latent}], + ) + inputs = comfyui_payload_to_branch_inputs( + video, + audio, + context, + minimax_payload={ + "keyframes": [{"resolved_frame_index": 0, "latent": kf_latent}], + "cond_video_latents": [kf_latent], + "visual_cond_noise_aug": 1.0, + }, + sample_sigmas=sample_sigmas, + timestep=timestep, + layout=serialize_comfyui_layout(layout), + ) + assert inputs["video_rows"].shape[0] == 4 + 2 * 2 * 2 + fk, branch = build_step_forward_kwargs(inputs, device=torch.device("cpu")) + assert set(fk) <= _FORWARD_SUPPORTED_KWARGS + assert branch.video_target_start == 4 + + +def test_uniform_denoise_mask_scales_video_timestep(): + video, audio, context, timestep, sample_sigmas = _t2va_tensors() + inputs = comfyui_payload_to_branch_inputs( + video, + audio, + context, + minimax_payload={}, + sample_sigmas=sample_sigmas, + timestep=timestep, + denoise_mask=torch.full((1, 1, 2, 4, 4), 0.5), + ) + sigma_v = float(timestep.reshape(-1)[0] / 1000.0) + # Text / pad keep the unmasked video timestep; only target rows remapped. + assert inputs["t_video"] == pytest.approx(1.0 - sigma_v) + assert inputs["video_rows_t"] is not None + assert float(inputs["video_rows_t"][0]) == pytest.approx(1.0 - 0.5 * sigma_v) + + +def test_per_frame_denoise_mask_overlays_only_video_targets(): + video, audio, context, timestep, sample_sigmas = _t2va_tensors() + mask = torch.ones(1, 1, 2, 4, 4) + mask[:, :, 0] = 1.0 + mask[:, :, 1] = 0.25 + inputs = comfyui_payload_to_branch_inputs( + video, + audio, + context, + minimax_payload={}, + sample_sigmas=sample_sigmas, + timestep=timestep, + denoise_mask=mask, + ) + fk, branch = build_step_forward_kwargs(inputs, device=torch.device("cpu")) + sigma_v = float(timestep.reshape(-1)[0] / 1000.0) + unique = fk["unique_timesteps"] + inverse = fk["inverse_indices"] + target_ts = unique[inverse[branch.img_target_seq_idx]] + frame0 = target_ts[:4] + frame1 = target_ts[4:] + assert torch.allclose( + frame0, torch.full_like(frame0, 1.0 - 1.0 * sigma_v), atol=1e-5 + ) + assert torch.allclose( + frame1, torch.full_like(frame1, 1.0 - 0.25 * sigma_v), atol=1e-5 + ) + + +def test_ref2va_image_forward_kwargs_are_supported(): + video, audio, context, timestep, sample_sigmas = _t2va_tensors() + ref_latent = torch.randn(1, 24, 1, 4, 4) + payload = { + "refs": [ + { + "kind": "image", + "latent_h": 4, + "latent_w": 4, + "latent": ref_latent, + } + ], + "text_token_tags": torch.ones(8, dtype=torch.long), + "visual_cond_noise_aug": 1.0, + } + inputs = comfyui_payload_to_branch_inputs( + video, + audio, + context, + minimax_payload=payload, + sample_sigmas=sample_sigmas, + timestep=timestep, + ) + assert inputs["video_rows"].shape[0] == 4 + 2 * 2 * 2 + assert inputs["audio_rows"].shape[0] == 6 + fk, branch = build_step_forward_kwargs(inputs, device=torch.device("cpu")) + assert set(fk) <= _FORWARD_SUPPORTED_KWARGS + assert branch.video_target_start == 4 + assert branch.audio_target_start == 0 + + from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import ( + MiniMaxH3ComfyUIRunState, + ) + + state = MiniMaxH3ComfyUIRunState( + branch=branch, + used=inputs["used"], + orig_video_shape=inputs["orig_video_shape"], + padded_video_shape=inputs["padded_video_shape"], + ) + v_video = torch.zeros(2 * 2 * 2, inputs["video_rows"].shape[1]) + v_audio = torch.zeros(inputs["audio_rows"].shape) + out_video, out_audio = pack_comfy_output(v_video, v_audio, state) + assert out_video.shape == video.shape + assert out_audio.shape == audio.shape + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="optional GPU smoke") +def test_optional_gpu_step_output_shapes(): + pytest.skip("no tiny H3 checkpoint on this machine; do not download weights") + + +# --- profile --- + +from sglang.multimodal_gen.runtime.pipelines.minimax_h3_pipeline import ( + MiniMaxH3Pipeline, +) +from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import ( + create_comfyui_pipeline_stages, +) +from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import ( + MiniMaxH3ComfyUIStepStage, +) + + +class _StubPipeline: + def __init__(self): + self._stage_name_mapping = {} + self._stages = [] + self.modules = {"transformer": object(), "scheduler": object()} + + def get_module(self, name): + return self.modules[name] + + def add_stage(self, stage, stage_name=None): + name = stage_name or stage.__class__.__name__ + self._stage_name_mapping[name] = stage + self._stages.append(stage) + return self + + +def test_create_comfyui_pipeline_stages_dispatches_to_pipeline_hook(): + called = {} + + class _Hooked(_StubPipeline): + def create_comfyui_stages(self, server_args): + called["server_args"] = server_args + + pipeline = _Hooked() + create_comfyui_pipeline_stages(pipeline, server_args="marker") + assert called["server_args"] == "marker" + assert pipeline._stage_name_mapping == {} + + +def test_h3_comfyui_stages_are_step_stage_only(monkeypatch): + created = {} + + def _fake_init(self, transformer): + self.transformer = transformer + created["transformer"] = transformer + + monkeypatch.setattr(MiniMaxH3ComfyUIStepStage, "__init__", _fake_init) + pipeline = _StubPipeline() + MiniMaxH3Pipeline.create_comfyui_stages(pipeline, server_args=None) + assert list(pipeline._stage_name_mapping) == ["MiniMaxH3ComfyUIStepStage"] + assert "DenoisingStage" not in pipeline._stage_name_mapping + assert "ComfyUILatentPreparationStage" not in pipeline._stage_name_mapping + assert created["transformer"] is pipeline.modules["transformer"] + + +def test_h3_profile_hook_installs_step_stage(monkeypatch): + def _fake_init(self, transformer): + self.transformer = transformer + + monkeypatch.setattr(MiniMaxH3ComfyUIStepStage, "__init__", _fake_init) + + class _H3(_StubPipeline): + create_comfyui_stages = MiniMaxH3Pipeline.create_comfyui_stages + + pipeline = _H3() + create_comfyui_pipeline_stages(pipeline, server_args=None) + assert list(pipeline._stage_name_mapping) == ["MiniMaxH3ComfyUIStepStage"] + assert "DenoisingStage" not in pipeline._stage_name_mapping + + +def test_comfyui_transformer_load_uses_gguf_override(monkeypatch): + from types import SimpleNamespace + + from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( + spec as comfyui_checkpoints_spec, + ) + + called = {} + + class _Spec: + dit_cls_name = "MiniMaxH3DiTModel" + param_names_mapping = {} + inherit_config_mapping = False + convert_weights = None + strict = True + + def build_dit_config(self, server_args): + arch = SimpleNamespace(param_names_mapping={}) + return SimpleNamespace(arch_config=arch) + + class _ModelCls: + param_names_mapping = {} + + class _Pipeline: + pipeline_name = "MiniMaxH3Pipeline" + model_path = "/tmp/minimax_h3_fl2va_bf16.safetensors" + modules = {"scheduler": "sched"} + + def get_module(self, name): + return self.modules[name] + + monkeypatch.setattr( + comfyui_checkpoints_spec, "get_comfyui_checkpoint_spec", lambda name: _Spec() + ) + monkeypatch.setattr( + comfyui_checkpoints_spec.ModelRegistry, + "resolve_model_cls", + lambda name: (_ModelCls, None), + ) + monkeypatch.setattr( + comfyui_checkpoints_spec, + "resolve_precision", + lambda *args, **kwargs: "bf16", + ) + monkeypatch.setattr( + "sglang.multimodal_gen.runtime.loader.transformer_load_utils.resolve_transformer_gguf_to_load", + lambda server_args, component: "/tmp/dit.gguf", + ) + + loaded = SimpleNamespace(parameters=lambda: []) + + def _fake_gguf(**kwargs): + called.update(kwargs) + return loaded + + monkeypatch.setattr( + comfyui_checkpoints_spec, "_load_comfyui_gguf_transformer", _fake_gguf + ) + + server_args = SimpleNamespace( + transformer_weights_path="/tmp/dit.gguf", + model_paths={}, + hsdp_replicate_dim=1, + hsdp_shard_dim=1, + pin_cpu_memory=True, + should_start_component_on_cpu=lambda name: False, + should_use_fsdp_for_component=lambda name: False, + ) + modules = comfyui_checkpoints_spec.load_comfyui_transformer( + _Pipeline(), server_args + ) + assert modules["transformer"] is loaded + assert called["gguf_file"] == "/tmp/dit.gguf" + + +# --- qkv load --- + +from types import SimpleNamespace + +import torch + +from sglang.multimodal_gen.configs.models.dits.minimax_h3 import MiniMaxH3DiTArchConfig +from sglang.multimodal_gen.runtime.distributed.parallel_state import ( + maybe_init_distributed_environment_and_model_parallel, + model_parallel_is_initialized, +) +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( + get_comfyui_checkpoint_spec, +) +from sglang.multimodal_gen.runtime.models.dits.minimax_h3 import ( + MiniMaxH3Attention, + _reorder_grouped_qkv_to_qkv, +) +from sglang.multimodal_gen.test.single_test_file.component_accuracy.utils import ( + ensure_distributed_env_defaults, +) + + +def _ensure_single_process_parallel_runtime() -> None: + if model_parallel_is_initialized(): + return + ensure_distributed_env_defaults() + maybe_init_distributed_environment_and_model_parallel(tp_size=1, sp_size=1) + + +def _concat_qkv_rows() -> torch.Tensor: + # Runtime / ComfyUI layout: [Q_all, K_all, V_all], 2 heads × head_dim 2. + return torch.tensor( + [0, 1, 2, 3, 10, 11, 12, 13, 20, 21, 22, 23], + dtype=torch.float32, + ).reshape(12, 1) + + +def test_grouped_reorder_is_not_identity_on_comfyui_concat_qkv() -> None: + concat = _concat_qkv_rows() + reordered = _reorder_grouped_qkv_to_qkv( + concat, + num_query_groups=2, + heads_per_group=1, + head_dim=2, + ) + assert not torch.equal(reordered, concat) + + +def test_comfyui_h3_spec_disables_grouped_qkv_reorder() -> None: + spec = get_comfyui_checkpoint_spec("MiniMaxH3Pipeline") + assert spec is not None + pipeline_config = SimpleNamespace(dit_config=None) + server_args = SimpleNamespace(pipeline_config=pipeline_config) + dit_config = spec.build_dit_config(server_args) + assert dit_config.arch_config.qkv_checkpoint_grouped is False + assert MiniMaxH3DiTArchConfig().qkv_checkpoint_grouped is True + + +def test_concat_qkv_attention_skips_grouped_weight_loader() -> None: + _ensure_single_process_parallel_runtime() + concat_arch = MiniMaxH3DiTArchConfig( + hidden_size=32, + num_attention_heads=2, + attention_head_dim=16, + qkv_checkpoint_grouped=False, + ) + concat_attn = MiniMaxH3Attention(concat_arch, None, prefix="blocks.0.attn") + concat_weight = concat_attn.qkv_proj.weight + assert getattr(concat_weight, "checkpoint_mapping_unsafe", False) is False + assert getattr(concat_weight, "rank_local_weight_transform", None) is None + + grouped_arch = MiniMaxH3DiTArchConfig( + hidden_size=32, + num_attention_heads=2, + attention_head_dim=16, + qkv_checkpoint_grouped=True, + ) + grouped_attn = MiniMaxH3Attention(grouped_arch, None, prefix="blocks.1.attn") + grouped_weight = grouped_attn.qkv_proj.weight + assert grouped_weight.checkpoint_mapping_unsafe is True + assert grouped_weight.rank_local_weight_transform is not None diff --git a/python/sglang/multimodal_gen/test/unit/test_comfyui_profile.py b/python/sglang/multimodal_gen/test/unit/test_comfyui_profile.py new file mode 100644 index 0000000000..f6d6dbcf15 --- /dev/null +++ b/python/sglang/multimodal_gen/test/unit/test_comfyui_profile.py @@ -0,0 +1,205 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Tests for the comfyui_mode profile of native pipelines. + +The existing ComfyUI tests under apps/ComfyUI_SGLDiffusion/test need a real +multi-GB checkpoint and two GPUs, so nothing here is covered in CI. These build +a tiny ComfyUI-format checkpoint instead, which is enough to exercise module +trimming, single-file loading, weight conversion, and stage construction. +""" + +import os +import tempfile + +import pytest +import torch +from safetensors.torch import save_file + +from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig +from sglang.multimodal_gen.registry import get_pipeline_class +from sglang.multimodal_gen.runtime.distributed.parallel_state import ( + maybe_init_distributed_environment_and_model_parallel, + model_parallel_is_initialized, +) +from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( + get_comfyui_checkpoint_spec, +) +from sglang.multimodal_gen.runtime.server_args import ServerArgs +from sglang.multimodal_gen.runtime.server_args.server_args import set_global_server_args +from sglang.multimodal_gen.test.single_test_file.component_accuracy.utils import ( + ensure_distributed_env_defaults, +) + +HEADS, HEAD_DIM, LAYERS, SINGLE_LAYERS = 4, 16, 2, 2 + + +def _ensure_single_process_parallel_runtime() -> None: + if model_parallel_is_initialized(): + return + ensure_distributed_env_defaults() + maybe_init_distributed_environment_and_model_parallel(tp_size=1, sp_size=1) + + +def _shrink(arch) -> None: + arch.num_attention_heads = HEADS + arch.attention_head_dim = HEAD_DIM + arch.num_layers = LAYERS + arch.num_single_layers = SINGLE_LAYERS + arch.guidance_embeds = True + + +def _write_comfyui_flux_checkpoint(path: str, config: FluxConfig) -> dict: + """Write a checkpoint using ComfyUI's parameter names and fused layouts.""" + arch = config.arch_config + hidden = arch.num_attention_heads * arch.attention_head_dim + mlp_hidden = int(hidden * getattr(arch, "mlp_ratio", 4.0)) + qkv = 3 * hidden + out_channels = arch.out_channels or arch.in_channels + patch = getattr(arch, "patch_size", 1) + + sd: dict[str, torch.Tensor] = {} + + def put(name: str, *shape: int) -> None: + sd[name] = torch.randn(*shape, dtype=torch.bfloat16) + + for b in range(arch.num_layers): + for attn in ("img_attn", "txt_attn"): + put(f"double_blocks.{b}.{attn}.qkv.weight", qkv, hidden) + put(f"double_blocks.{b}.{attn}.qkv.bias", qkv) + put(f"double_blocks.{b}.{attn}.proj.weight", hidden, hidden) + put(f"double_blocks.{b}.{attn}.proj.bias", hidden) + put( + f"double_blocks.{b}.{attn}.norm.query_norm.scale", + arch.attention_head_dim, + ) + put( + f"double_blocks.{b}.{attn}.norm.key_norm.scale", arch.attention_head_dim + ) + for mlp in ("img_mlp", "txt_mlp"): + put(f"double_blocks.{b}.{mlp}.0.weight", mlp_hidden, hidden) + put(f"double_blocks.{b}.{mlp}.0.bias", mlp_hidden) + put(f"double_blocks.{b}.{mlp}.2.weight", hidden, mlp_hidden) + put(f"double_blocks.{b}.{mlp}.2.bias", hidden) + for mod in ("img_mod", "txt_mod"): + put(f"double_blocks.{b}.{mod}.lin.weight", 6 * hidden, hidden) + put(f"double_blocks.{b}.{mod}.lin.bias", 6 * hidden) + + for b in range(arch.num_single_layers): + put(f"single_blocks.{b}.linear1.weight", qkv + mlp_hidden, hidden) + put(f"single_blocks.{b}.linear1.bias", qkv + mlp_hidden) + put(f"single_blocks.{b}.linear2.weight", hidden, hidden + mlp_hidden) + put(f"single_blocks.{b}.linear2.bias", hidden) + put(f"single_blocks.{b}.norm.query_norm.scale", arch.attention_head_dim) + put(f"single_blocks.{b}.norm.key_norm.scale", arch.attention_head_dim) + put(f"single_blocks.{b}.modulation.lin.weight", 3 * hidden, hidden) + put(f"single_blocks.{b}.modulation.lin.bias", 3 * hidden) + + for stem, in_dim in ( + ("time_in", 256), + ("vector_in", arch.pooled_projection_dim), + ("guidance_in", 256), + ): + put(f"{stem}.in_layer.weight", hidden, in_dim) + put(f"{stem}.in_layer.bias", hidden) + put(f"{stem}.out_layer.weight", hidden, hidden) + put(f"{stem}.out_layer.bias", hidden) + + put("txt_in.weight", hidden, arch.joint_attention_dim) + put("txt_in.bias", hidden) + put("img_in.weight", hidden, arch.in_channels) + put("img_in.bias", hidden) + put("final_layer.linear.weight", patch * patch * out_channels, hidden) + put("final_layer.linear.bias", patch * patch * out_channels) + put("final_layer.adaLN_modulation.1.weight", 2 * hidden, hidden) + put("final_layer.adaLN_modulation.1.bias", 2 * hidden) + + save_file(sd, path) + return sd + + +@pytest.fixture(scope="module") +def comfyui_flux_pipeline(): + if not torch.cuda.is_available(): + pytest.skip("requires CUDA") + + tmpdir = tempfile.mkdtemp(prefix="comfyui_flux_") + checkpoint = os.path.join(tmpdir, "flux_comfyui.safetensors") + + # The file has to exist before ServerArgs resolves the single-file path. + seed_config = FluxConfig() + _shrink(seed_config.arch_config) + state_dict = _write_comfyui_flux_checkpoint(checkpoint, seed_config) + + server_args = ServerArgs.from_kwargs( + model_path=checkpoint, + pipeline_class_name="FluxPipeline", + comfyui_mode=True, + num_gpus=1, + ) + _shrink(server_args.pipeline_config.dit_config.arch_config) + set_global_server_args(server_args) + _ensure_single_process_parallel_runtime() + + pipeline = get_pipeline_class("FluxPipeline")( + model_path=checkpoint, server_args=server_args + ) + return pipeline, state_dict + + +def test_specs_cover_every_comfyui_supported_pipeline(): + for pipeline_name in ( + "FluxPipeline", + "ZImagePipeline", + "QwenImagePipeline", + "QwenImageEditPlusPipeline", + ): + spec = get_comfyui_checkpoint_spec(pipeline_name) + assert spec is not None, f"{pipeline_name} has no ComfyUI checkpoint spec" + assert get_pipeline_class(pipeline_name) is not None + assert get_pipeline_class(pipeline_name).pipeline_config_cls is not None + + +def test_comfyui_mode_trims_pipeline_to_a_dit_forward_service(comfyui_flux_pipeline): + pipeline, _ = comfyui_flux_pipeline + + assert pipeline.required_config_modules == ["transformer", "scheduler"] + assert ( + type(pipeline.get_module("scheduler")).__name__ == "ComfyUIPassThroughScheduler" + ) + assert list(pipeline._stage_name_mapping) == [ + "ComfyUILatentPreparationStage", + "DenoisingStage", + ] + + +def test_comfyui_checkpoint_fully_populates_the_transformer(comfyui_flux_pipeline): + pipeline, _ = comfyui_flux_pipeline + transformer = pipeline.get_module("transformer") + + unloaded = [name for name, p in transformer.named_parameters() if p.is_meta] + assert not unloaded, f"parameters never received checkpoint weights: {unloaded[:5]}" + + +def test_fused_qkv_is_split_into_separate_projections(comfyui_flux_pipeline): + pipeline, state_dict = comfyui_flux_pipeline + params = dict(pipeline.get_module("transformer").named_parameters()) + device = next(iter(params.values())).device + hidden = HEADS * HEAD_DIM + + fused = state_dict["double_blocks.0.img_attn.qkv.weight"].to(device, torch.bfloat16) + for offset, proj in enumerate(("to_q", "to_k", "to_v")): + expected = fused[offset * hidden : (offset + 1) * hidden] + actual = params[f"transformer_blocks.0.attn.{proj}.weight"] + assert torch.equal(actual, expected), f"{proj} does not match the fused slice" + + +def test_adaln_modulation_scale_and_shift_are_swapped(comfyui_flux_pipeline): + pipeline, state_dict = comfyui_flux_pipeline + params = dict(pipeline.get_module("transformer").named_parameters()) + actual = params["norm_out.linear.weight"] + + source = state_dict["final_layer.adaLN_modulation.1.weight"].to( + actual.device, torch.bfloat16 + ) + half = source.shape[0] // 2 + # ComfyUI emits [shift, scale]; AdaLayerNormContinuous expects [scale, shift]. + assert torch.equal(actual, torch.cat([source[half:], source[:half]], dim=0)) diff --git a/python/sglang/multimodal_gen/test/unit/test_comfyui_session.py b/python/sglang/multimodal_gen/test/unit/test_comfyui_session.py new file mode 100644 index 0000000000..60f2fec668 --- /dev/null +++ b/python/sglang/multimodal_gen/test/unit/test_comfyui_session.py @@ -0,0 +1,229 @@ +# SPDX-License-Identifier: Apache-2.0 + +import torch + +from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import ( + bind_comfyui_session, + get_run_state, + initialize_comfyui_pipeline, + release_comfyui_session, + release_run_state, + set_run_state, +) + + +class _Req: + def __init__(self): + self.extra = {} + self.prompt_embeds = [] + self.prompt_seq_lens = None + self.pooled_embeds = [] + + +def test_latent_prep_verify_input_restores_cached_embeds() -> None: + """Later ComfyUI steps omit embeds; verify_input must restore before checks.""" + from types import SimpleNamespace + + from sglang.multimodal_gen.runtime.pipelines_core.stages.comfyui_latent_preparation import ( + ComfyUILatentPreparationStage, + ) + + sid = "run-verify" + embeds = [torch.ones(2, 4)] + first = _Req() + first.extra["comfyui_session_id"] = sid + first.prompt_embeds = embeds + first.prompt_seq_lens = [[2]] + bind_comfyui_session(first) + + batch = SimpleNamespace( + extra={"comfyui_session_id": sid}, + prompt_embeds=[], + prompt=" ", + num_outputs_per_prompt=1, + generator=torch.Generator("cpu"), + num_frames=1, + height=64, + width=64, + latents=None, + prompt_seq_lens=None, + pooled_embeds=None, + negative_prompt_embeds=None, + negative_prompt_seq_lens=None, + neg_pooled_embeds=None, + image_latent=None, + vae_image_sizes=None, + prompt_attention_mask=None, + negative_attention_mask=None, + prompt_embeds_mask=None, + negative_prompt_embeds_mask=None, + ) + stage = ComfyUILatentPreparationStage(scheduler=None, transformer=None) + result = stage.verify_input(batch, server_args=None) + assert result.is_valid() + assert torch.equal(batch.prompt_embeds[0], embeds[0]) + release_comfyui_session(sid) + + +def test_session_restores_conditioning_on_later_steps() -> None: + sid = "run-1" + first = _Req() + first.extra["comfyui_session_id"] = sid + first.prompt_embeds = [torch.ones(4, 8)] + first.prompt_seq_lens = [[4]] + bind_comfyui_session(first) + + second = _Req() + second.extra["comfyui_session_id"] = sid + bind_comfyui_session(second) + + assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0]) + assert second.prompt_seq_lens == [[4]] + release_comfyui_session(sid) + + third = _Req() + third.extra["comfyui_session_id"] = sid + bind_comfyui_session(third) + assert third.prompt_embeds == [] + + +def test_release_run_state_keeps_conditioning(): + sid = "run-keep-cond" + first = _Req() + first.extra = {"comfyui_session_id": sid, "model_payload": {"audio_scale": 0.5}} + first.prompt_embeds = [torch.ones(2, 4)] + bind_comfyui_session(first) + set_run_state(first, {"branch": "alive"}) + release_run_state(sid) + assert get_run_state(first) is None + second = _Req() + second.extra = {"comfyui_session_id": sid} + bind_comfyui_session(second) + assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0]) + assert second.extra["model_payload"]["audio_scale"] == 0.5 + release_comfyui_session(sid) + + +def test_session_restores_extra_keys_on_later_steps() -> None: + sid = "run-extra" + payload = {"audio_scale": 0.5} + sigmas = torch.tensor([1.0, 0.5, 0.0]) + first = _Req() + first.extra = { + "comfyui_session_id": sid, + "model_payload": payload, + "sample_sigmas": sigmas, + } + first.prompt_embeds = [torch.ones(3, 4)] + bind_comfyui_session(first) + + second = _Req() + second.extra = {"comfyui_session_id": sid} + bind_comfyui_session(second) + assert second.extra["model_payload"] is payload + assert torch.equal(second.extra["sample_sigmas"], sigmas) + assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0]) + release_comfyui_session(sid) + + +def test_new_run_id_evicts_previous_run_for_same_executor() -> None: + first = _Req() + first.extra = {"comfyui_session_id": "exec1:1", "h3_layout": {"signature": (1, 2)}} + first.prompt_embeds = [torch.ones(2, 4)] + bind_comfyui_session(first) + set_run_state(first, {"branch": "old"}) + + second = _Req() + second.extra = {"comfyui_session_id": "exec1:2"} + bind_comfyui_session(second) + assert "h3_layout" not in second.extra + assert second.prompt_embeds == [] + old = _Req() + old.extra = {"comfyui_session_id": "exec1:1"} + assert get_run_state(old) is None + release_comfyui_session("exec1:2") + + +def test_fingerprint_mismatch_does_not_restore_extras() -> None: + sid = "exec2:1" + first = _Req() + first.extra = { + "comfyui_session_id": sid, + "comfyui_cache_fp": {"spatial": (5, 15, 27)}, + "h3_layout": {"signature": (8, 5, 15, 27, 4)}, + } + first.prompt_embeds = [torch.ones(2, 4)] + bind_comfyui_session(first) + + second = _Req() + second.extra = { + "comfyui_session_id": sid, + "comfyui_cache_fp": {"spatial": (5, 30, 54)}, + } + bind_comfyui_session(second) + assert "h3_layout" not in second.extra + assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0]) + release_comfyui_session(sid) + + +def test_cond_keys_keep_positive_and_negative_apart() -> None: + sid = "exec3:1" + pos = _Req() + pos.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "pos"} + pos.prompt_embeds = [torch.ones(2, 4)] + bind_comfyui_session(pos) + neg = _Req() + neg.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "neg"} + neg.prompt_embeds = [torch.zeros(2, 4)] + bind_comfyui_session(neg) + + later_pos = _Req() + later_pos.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "pos"} + bind_comfyui_session(later_pos) + later_neg = _Req() + later_neg.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "neg"} + bind_comfyui_session(later_neg) + assert torch.equal(later_pos.prompt_embeds[0], pos.prompt_embeds[0]) + assert torch.equal(later_neg.prompt_embeds[0], neg.prompt_embeds[0]) + release_comfyui_session(sid) + + +class _FakePipeline: + def __init__(self): + self.modules = {} + + +def _init_vae_geometry(pipeline_config): + from types import SimpleNamespace + + initialize_comfyui_pipeline( + _FakePipeline(), + SimpleNamespace(pipeline_config=pipeline_config, comfyui_mode=True), + ) + return pipeline_config.vae_config.arch_config + + +def test_comfyui_mode_derives_flux_vae_scale_factor() -> None: + from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig + + arch = _init_vae_geometry(FluxPipelineConfig()) + assert arch.vae_scale_factor == 8 + + +def test_comfyui_mode_derives_qwen_vae_scale_factor() -> None: + from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import ( + QwenImagePipelineConfig, + ) + + arch = _init_vae_geometry(QwenImagePipelineConfig()) + assert arch.vae_scale_factor == 8 + + +def test_comfyui_mode_h3_vae_post_init_without_latent_stats() -> None: + from sglang.multimodal_gen.configs.pipeline_configs.minimax_h3 import ( + MiniMaxH3PipelineConfig, + ) + + arch = _init_vae_geometry(MiniMaxH3PipelineConfig()) + assert arch.latents_mean is None + assert arch.latents_std is None diff --git a/python/sglang/multimodal_gen/test/unit/test_component_loader_identity.py b/python/sglang/multimodal_gen/test/unit/test_component_loader_identity.py index ca031421c5..8062c8e6bf 100644 --- a/python/sglang/multimodal_gen/test/unit/test_component_loader_identity.py +++ b/python/sglang/multimodal_gen/test/unit/test_component_loader_identity.py @@ -92,6 +92,7 @@ class TestComponentLoaderIdentity(unittest.TestCase): component_paths={}, component_direct_gpu_weight_loading=set(), resolve_component_attention_backend=lambda *_names: (None, None), + comfyui_mode=False, ) with patch.object( diff --git a/python/sglang/multimodal_gen/test/unit/test_fsdp_load.py b/python/sglang/multimodal_gen/test/unit/test_fsdp_load.py index 5cd84a4b59..4cc246214b 100644 --- a/python/sglang/multimodal_gen/test/unit/test_fsdp_load.py +++ b/python/sglang/multimodal_gen/test/unit/test_fsdp_load.py @@ -443,6 +443,46 @@ class TestRankLocalSafetensorsRead(unittest.TestCase): rank_local_checkpoint.tp_local_shape(sources, 0, 2), (4, 4) ) + def test_reads_tp_local_fused_merged_column_per_logical_matrix(self): + with tempfile.TemporaryDirectory() as temp_dir: + file_path = str(Path(temp_dir) / "model.safetensors") + q = torch.arange(16, dtype=torch.bfloat16).reshape(4, 4) + k = torch.arange(16, 32, dtype=torch.bfloat16).reshape(4, 4) + v = torch.arange(32, 48, dtype=torch.bfloat16).reshape(4, 4) + fused = torch.cat((q, k, v), dim=0) + save_file({"qkv": fused}, file_path) + sources = [self._source(file_path, "qkv", (12, 4))] + + with safe_open(file_path, framework="pt", device="cpu") as handle: + naive = rank_local_checkpoint.read_tp_local_tensor( + sources, + {file_path: handle}, + shard_dim=0, + tp_rank=0, + tp_size=2, + ) + rank0 = rank_local_checkpoint.read_tp_local_tensor( + sources, + {file_path: handle}, + shard_dim=0, + tp_rank=0, + tp_size=2, + output_sizes=[4, 4, 4], + ) + rank1 = rank_local_checkpoint.read_tp_local_tensor( + sources, + {file_path: handle}, + shard_dim=0, + tp_rank=1, + tp_size=2, + output_sizes=[4, 4, 4], + ) + + torch.testing.assert_close(naive, fused[:6]) + torch.testing.assert_close(rank0, torch.cat((q[:2], k[:2], v[:2]))) + torch.testing.assert_close(rank1, torch.cat((q[2:], k[2:], v[2:]))) + self.assertFalse(torch.equal(rank0, naive)) + def test_reads_tp_local_merged_row_slice(self): with tempfile.TemporaryDirectory() as temp_dir: file_path = str(Path(temp_dir) / "model.safetensors") diff --git a/python/sglang/multimodal_gen/test/unit/test_ipc_cuda.py b/python/sglang/multimodal_gen/test/unit/test_ipc_cuda.py new file mode 100644 index 0000000000..448a912746 --- /dev/null +++ b/python/sglang/multimodal_gen/test/unit/test_ipc_cuda.py @@ -0,0 +1,133 @@ +# SPDX-License-Identifier: Apache-2.0 + +import pickle +from dataclasses import dataclass + +import pytest +import torch + +from sglang.multimodal_gen.runtime.distributed.ipc_cuda import ( + CudaIpcRef, + attach_cuda_tensors, + detach_cuda_tensors, + materialize_cuda_refs, + spill_cuda_tensors, +) + + +def _require_cuda() -> None: + if not torch.cuda.is_available(): + pytest.skip("CUDA is required for CUDA IPC tests") + + +def test_pickle_copies_cuda_tensor_through_host() -> None: + _require_cuda() + tensor = torch.ones(16, 90, 160, device="cuda", dtype=torch.bfloat16) + assert len(pickle.dumps(tensor)) > tensor.nbytes + + +def test_ipc_handle_pickle_is_much_smaller_than_tensor() -> None: + _require_cuda() + tensor = torch.ones(16, 90, 160, device="cuda", dtype=torch.bfloat16) + ref = CudaIpcRef.from_tensor(tensor) + assert len(pickle.dumps(ref)) < 2048 + rebuilt = ref.materialize() + assert rebuilt.device.type == "cuda" + assert rebuilt.shape == tensor.shape + assert rebuilt.dtype == tensor.dtype + assert torch.equal(rebuilt, tensor) + + +def test_spill_does_not_mutate_caller_req_tensors() -> None: + _require_cuda() + + @dataclass + class _Holder: + latents: torch.Tensor + prompt_embeds: list + + latents = torch.arange(24, device="cuda", dtype=torch.float32).reshape(2, 3, 4) + embeds = [torch.ones(3, 5, device="cuda", dtype=torch.float32)] + holder = _Holder(latents=latents, prompt_embeds=embeds) + + spilled = spill_cuda_tensors(holder) + assert isinstance(spilled.latents, CudaIpcRef) + assert isinstance(spilled.prompt_embeds[0], CudaIpcRef) + assert holder.latents is latents + assert holder.prompt_embeds[0] is embeds[0] + + restored = materialize_cuda_refs(spilled) + assert torch.equal(restored.latents, latents) + assert torch.equal(restored.prompt_embeds[0], embeds[0]) + + +def test_detach_attach_keeps_non_cuda_fields() -> None: + _require_cuda() + + @dataclass + class _Holder: + extra: dict + vae_image_sizes: list + image_embeds: list + latents: torch.Tensor + prompt_embeds: list + + latents = torch.arange(24, device="cuda", dtype=torch.float32).reshape(2, 3, 4) + embeds = [torch.ones(2, 8, device="cuda", dtype=torch.float32)] + holder = _Holder( + extra={"comfyui_session_id": "run-1"}, + vae_image_sizes=[(64, 48)], + image_embeds=[], + latents=latents, + prompt_embeds=embeds, + ) + + skeleton, tensors = detach_cuda_tensors(holder) + assert holder.latents is latents + assert skeleton.extra["comfyui_session_id"] == "run-1" + assert skeleton.vae_image_sizes == [(64, 48)] + assert skeleton.image_embeds == [] + assert skeleton.latents is None + + restored = attach_cuda_tensors(skeleton, tensors) + assert restored.extra["comfyui_session_id"] == "run-1" + assert restored.vae_image_sizes == [(64, 48)] + assert restored.image_embeds == [] + assert restored.latents.device.type == "cuda" + assert torch.equal(restored.latents, latents) + assert torch.equal(restored.prompt_embeds[0], embeds[0]) + + +def test_cross_process_ipc_roundtrip() -> None: + _require_cuda() + ctx = torch.multiprocessing.get_context("spawn") + tensor = torch.arange(64, device="cuda", dtype=torch.float32).reshape(4, 16) + ref = CudaIpcRef.from_tensor(tensor) + queue = ctx.Queue() + proc = ctx.Process(target=_cross_process_child, args=(queue, pickle.dumps(ref))) + proc.start() + try: + shape, dtype, first, last = queue.get(timeout=30) + finally: + proc.join(timeout=30) + if proc.is_alive(): + proc.kill() + proc.join() + assert proc.exitcode == 0 + assert shape == (4, 16) + assert dtype == "torch.float32" + assert first == 0.0 + assert last == 63.0 + + +def _cross_process_child(queue, payload: bytes) -> None: + ref = pickle.loads(payload) + rebuilt = ref.materialize() + queue.put( + ( + tuple(rebuilt.shape), + str(rebuilt.dtype), + float(rebuilt.reshape(-1)[0]), + float(rebuilt.reshape(-1)[-1]), + ) + )