[diffusion] feat: add MiniMax-H3 to ComfyUI integrated mode (#35990)

This commit is contained in:
WenhaoZhang
2026-09-27 17:58:54 +08:00
committed by GitHub
parent ae47bcd4da
commit f1e62e3a2e
62 changed files with 5848 additions and 1967 deletions
+1 -1
View File
@@ -11,7 +11,7 @@ SGLang diffusion features an end-to-end unified pipeline for accelerating diffus
SGLang Diffusion has the following features:
- Broad model support: Wan, FastWan, FLUX, Qwen-Image / Qwen-Image 2.1, LongCat-Image, Z-Image, Anima, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3/LTX-2.5, MiniMax-H3, FastH3, VDN-H3, LingBot Video MoE, LingBot World, SANA-Video/SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more
- Fast inference speed: empowered by optimized `sgl-kernel` kernels, scheduler/runtime improvements, caching acceleration, and native diffusion hot-path optimizations
- Ease of use: OpenAI-compatible api, CLI, and python sdk support
- Ease of use: OpenAI-compatible api, CLI, python sdk, and a [ComfyUI plugin](apps/ComfyUI_SGLDiffusion/README.md)
- Multi-platform support:
- NVIDIA GPUs (H100, H200, A100, B200, 4090, 5090)
- AMD GPUs (MI300X, MI325X, MI355X)
@@ -1,6 +1,14 @@
# ComfyUI SGLDiffusion Plugin
A ComfyUI plugin for integrating with SGLang Diffusion server, supporting image and video generation capabilities.
A ComfyUI plugin for SGLang Diffusion. Server mode talks to a standalone HTTP
server. Integrated mode keeps ComfyUI's CLIP / VAE / sampler loop and uses
SGLang only as a per-step DiT forward.
Integrated mode no longer ships dedicated `comfyui_*` pipelines. It starts the
native Flux / Qwen-Image / Z-Image / MiniMax-H3 pipeline under `--comfyui-mode`,
loads a single-file ComfyUI `.safetensors` (or GGUF overlay) through a
checkpoint spec, and translates each `apply_model` call through a small
per-model adapter.
## Installation
@@ -16,7 +24,7 @@ The plugin supports two modes of operation: **Server Mode** (via HTTP API) and *
- **Z-Image**: High-speed image generation models (e.g., `Z-Image-Turbo`)
- **FLUX**: State-of-the-art text-to-image models (e.g., `FLUX.1-dev`)
- **Qwen-Image**: Multi-modal image generation models (e.g., `Qwen-Image`,`Qwen-Image-2512`). *Note: Image editing support is currently experimental and may have some issues.*
- **MiniMax-H3**: Joint video-and-audio generation, server mode only (`SGLDiffusion Generate H3`)
- **MiniMax-H3**: Joint video-and-audio DiT (`model_type=minimax_h3`). Integrated mode: T2V / I2VA / FL2VA use an `fl2va` checkpoint; R2V needs `ref2va`. CLIP and VAE stay in ComfyUI. Server mode uses `SGLDiffusion Generate MiniMax-H3`.
### Mode 1: Server Mode (HTTP API)
Connect to a standalone SGLang Diffusion server.
@@ -26,14 +34,17 @@ Connect to a standalone SGLang Diffusion server.
3. **Generate Content**:
- `SGLDiffusion Generate Image`: For text-to-image and image editing.
- `SGLDiffusion Generate Video`: For text-to-video and image-to-video.
- `SGLDiffusion Generate MiniMax-H3`: For joint video-and-audio on a standalone server.
4. **LoRA Support**: Use `SGLDiffusion Server Set LoRA` and `SGLDiffusion Server Unset LoRA`.
### Mode 2: Integrated Mode (Tight Integration)
Leverage SGLang's high-performance sampling directly within ComfyUI while using ComfyUI's front-end nodes (CLIP, VAE, etc.).
ComfyUI keeps CLIP, VAE, and the sampler loop. SGLang loads only the DiT and
runs one forward per sampler step (pass-through scheduler, no text encode /
decode on the worker).
1. **Load Model**: Use the `SGLDiffusion UNET Loader` node to load your diffusion model.
2. **Configure Options**: Use the `SGLDiffusion Options` node to set runtime parameters like `num_gpus`, `tp_size`, `model_type`, or `enable_torch_compile`.
3. **Sample**: Connect the loaded model to standard ComfyUI samplers. SGLang will handle the sampling process efficiently.
3. **Sample**: Connect the loaded model to standard ComfyUI samplers. Each step is packed by a model adapter and sent to the SGLang scheduler.
4. **LoRA Support**: Use the `SGLDiffusion LoRA Loader` for native LoRA integration.
## Adding a Model
@@ -74,6 +85,9 @@ Reference workflow files are provided in the `workflows/` directory:
- **`z-image_sgld.json`**: High-speed image generation using Z-Image.
- **`sgld_text2img.json`**: Server-mode text-to-image generation with LoRA support.
- **`sgld_image2video.json`**: Server-mode image-to-video generation.
- **`minimax_h3_t2v_sgld.json`**: MiniMax-H3 T2V / I2VA / FL2VA (`fl2va` DiT).
- **`minimax_h3_r2v_sgld.json`**: MiniMax-H3 reference-to-video (`ref2va` DiT).
- **`minimax_h3_t2v_sgld_upscaler.json`**: H3 two-pass latent upscale (low-res then 3D ×2 refine).
For other workflows supporting the models, you can easily use SGLang by replacing the official `UNET Loader` node with the `SGLDUNETLoader` node. Similarly, for LoRA support, replace the official LoRA loader with the `SGLDiffusion LoRA Loader`.
@@ -85,4 +99,45 @@ To use these workflows:
## Current Implementation
This plugin provides a high-performance backend for diffusion models in ComfyUI. By leveraging SGLang's optimized kernels and parallelization techniques (Tensor Parallelism, TeaCache, etc.), it significantly accelerates the sampling process, especially for large models like FLUX.
SGLang's optimized kernels and parallelism (TP / SP, compile, cache) run on the
DiT only. Text encoding and VAE stay in ComfyUI.
## Architecture
```mermaid
flowchart LR
subgraph comfy [ComfyUI process]
CLIP[CLIP / text encode]
VAE[VAE]
SAMPLER[Sampler loop]
EXEC["SGLDiffusionExecutor"]
ADAPT["Model adapter<br/>pack / unpack"]
CLIP --> SAMPLER
SAMPLER --> EXEC --> ADAPT
end
ADAPT -->|CUDA IPC spill| R0
subgraph sgl [SGLang]
R0[Rank-0 scheduler]
R0 -->|comfyui_mode and multi-rank| NCCL["Detach CUDA tensors<br/>NCCL broadcast"]
R0 -->|otherwise| PYO["Original SP / CFG / TP<br/>broadcast_pyobj"]
NCCL --> PIPE
PYO --> PIPE
PIPE["Native pipeline<br/>--comfyui-mode"]
SPEC["Checkpoint spec<br/>single .safetensors"] --> PIPE
PIPE --> STAGE["Latent prep + session cache<br/>+ DenoisingStage"]
end
STAGE -->|noise_pred IPC| EXEC
STAGE --> VAE
```
Per sampler step:
1. The adapter turns ComfyUI `apply_model` tensors into an SGLang `Req`.
2. Local ZMQ pickle replaces CUDA tensors with IPC handles so latents stay on GPU.
3. Rank 0 materializes the handles. Multi-rank `--comfyui-mode` then detaches CUDA tensors and broadcasts them over NCCL; the general SP / CFG / TP path is still the original whole-list `broadcast_pyobj`.
4. The worker pipeline is the native model class with modules trimmed to `transformer` + pass-through scheduler. A single-file checkpoint goes through `comfyui_checkpoints`. After the first step, conditioning stays in a worker session; later steps send latents and the timestep.
Adding a model means a checkpoint spec plus a `ComfyUIModelAdapter`. There is no extra ComfyUI pipeline class.
@@ -4,7 +4,6 @@ Provides generator, model patcher, and server API client.
"""
from .generator import SGLDiffusionGenerator
from .model_patcher import SGLDModelPatcher
from .server_api import SGLDiffusionServerAPI
__all__ = [
@@ -12,3 +11,12 @@ __all__ = [
"SGLDModelPatcher",
"SGLDiffusionServerAPI",
]
def __getattr__(name):
# ModelPatcher subclasses ComfyUI and is only needed when a graph loads a DiT.
if name == "SGLDModelPatcher":
from .model_patcher import SGLDModelPatcher
return SGLDModelPatcher
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
@@ -2,20 +2,103 @@
Generator for SGLang Diffusion ComfyUI integration.
"""
import atexit
import logging
import os
import psutil
from comfy import model_detection, model_management
from comfy.utils import (
calculate_parameters,
load_torch_file,
state_dict_prefix_replace,
unet_to_diffusers,
)
from ..executors.flux import FluxExecutor
from ..executors.minimax_h3 import MiniMaxH3Executor
from ..executors.zimage import ZImageExecutor
logger = logging.getLogger(__name__)
class _HeaderTensor:
"""Shape-only stand-in so detect_unet_config does not load 66GB weights."""
def __init__(self, shape):
import torch
self.shape = torch.Size(shape)
def numel(self):
n = 1
for dim in self.shape:
n *= int(dim)
return n
nelement = numel
def _looks_like_gguf(path: str) -> bool:
if not path:
return False
return path.lower().endswith(".gguf") or (
":" in path and path.count("/") == 1 and not os.path.isabs(path)
)
def _h3_detect_companion(gguf_ref: str) -> str:
"""ComfyUI detect needs H3 safetensors keys; GGUF only overrides the DiT."""
name = os.path.basename(gguf_ref).lower()
prefer = []
if "ref2va" in name:
prefer.append("minimax_h3_ref2va_bf16.safetensors")
prefer.extend(
[
"minimax_h3_fl2va_bf16.safetensors",
"minimax_h3_ref2va_bf16.safetensors",
]
)
folders = (
[os.path.dirname(os.path.abspath(gguf_ref))]
if os.path.dirname(gguf_ref)
else []
)
try:
import folder_paths
folders.extend(folder_paths.get_folder_paths("diffusion_models"))
except Exception:
pass
seen: set[str] = set()
for folder in folders:
if not folder:
continue
for candidate in prefer:
path = os.path.join(folder, candidate)
if path in seen:
continue
seen.add(path)
if os.path.isfile(path):
return path
raise ValueError(
"GGUF transformer needs a MiniMax-H3 BF16 safetensors in "
"models/diffusion_models for ComfyUI architecture detect "
f"(looked for {prefer}). Keep --model-path / unet_name on the "
"BF16 file and pass the GGUF via transformer_weights_path."
)
def _load_state_dict_for_detection(model_path: str):
if _looks_like_gguf(model_path):
raise ValueError(
f"Cannot detect architecture from GGUF {model_path!r}. "
"Use a MiniMax-H3 BF16 safetensors for detect."
)
if model_path.endswith(".safetensors"):
from safetensors import safe_open
sd = {}
with safe_open(model_path, framework="pt", device="cpu") as handle:
for key in handle.keys():
sd[key] = _HeaderTensor(handle.get_slice(key).get_shape())
return sd
from comfy.utils import load_torch_file
return load_torch_file(model_path)
try:
from sglang.multimodal_gen import DiffGenerator
except ImportError:
@@ -23,39 +106,78 @@ except ImportError:
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
)
from ..executors import (
FluxExecutor,
QwenImageEditExecutor,
QwenImageExecutor,
ZImageExecutor,
)
from .model_patcher import SGLDModelPatcher
def _load_executor_classes():
"""Qwen adapters import ComfyUI. Keep them optional so CI can load the rest."""
classes = [FluxExecutor, ZImageExecutor, MiniMaxH3Executor]
try:
from ..executors.qwen_image import QwenImageEditExecutor, QwenImageExecutor
except ModuleNotFoundError as exc:
missing = exc.name or ""
if not missing.startswith("comfy"):
raise
logger.info("ComfyUI is not installed; Qwen image executors are unavailable")
else:
classes.extend([QwenImageExecutor, QwenImageEditExecutor])
return tuple(classes)
_EXECUTOR_CLASSES = _load_executor_classes()
_SHARED = None
_ATEXIT_REGISTERED = False
class SGLDiffusionGenerator:
"""Generator for SGLang Diffusion models in ComfyUI."""
"""Process-wide owner of the one live SGLD worker.
ComfyUI keeps one ``SGLDUNETLoader`` per node and may reuse a cached
``MODEL`` after another graph has already replaced the worker.
``sync_scheduler_client`` is also a process singleton, so a second
``DiffGenerator`` cannot coexist. All loaders share this object; a
stale executor rebinds itself on the next forward.
"""
def __init__(self):
self.model_path = None
self.generator = None
self.executor = None
self.last_options = None
self._patcher = None
self.pipeline_class_dict = {
"flux": "ComfyUIFluxPipeline",
"lumina2": "ComfyUIZImagePipeline", # zimage
"qwen_image": "ComfyUIQwenImagePipeline",
"qwen_image_edit": "ComfyUIQwenImageEditPipeline",
}
self.executor_class_dict = {
"flux": FluxExecutor,
"lumina2": ZImageExecutor,
"qwen_image": QwenImageExecutor,
"qwen_image_edit": QwenImageEditExecutor,
}
# Native pipelines, run under comfyui_mode as a DiT-only forward service.
self.pipeline_class_dict = {}
self.executor_class_dict = {}
for executor_cls in _EXECUTOR_CLASSES:
for model_type in executor_cls.adapter_cls.model_types:
self.executor_class_dict[model_type] = executor_cls
self.pipeline_class_dict[model_type] = (
executor_cls.adapter_cls.pipeline_class_name
)
def __del__(self):
self.close_generator()
@classmethod
def shared(cls):
global _SHARED, _ATEXIT_REGISTERED
if _SHARED is None:
_SHARED = cls()
if not _ATEXIT_REGISTERED:
atexit.register(cls._atexit_close)
_ATEXIT_REGISTERED = True
return _SHARED
@classmethod
def reset_shared(cls):
"""Test helper: drop the process singleton after closing it."""
global _SHARED
if _SHARED is not None:
_SHARED.close_generator()
_SHARED = None
@classmethod
def _atexit_close(cls):
if _SHARED is not None:
_SHARED.close_generator()
def init_generator(
self, model_path: str, pipeline_class_name: str, kwargs: dict = None
@@ -66,7 +188,13 @@ class SGLDiffusionGenerator:
if kwargs is None:
kwargs = {}
# Set comfyui_mode for ComfyUI integration
kwargs = dict(kwargs)
kwargs["comfyui_mode"] = True
# ComfyUI already keeps CLIP/VAE in the parent process. Auto image
# policy otherwise sets dit_cpu_offload=True and every sampler step
# reloads the DiT from CPU.
kwargs.setdefault("dit_cpu_offload", False)
kwargs = self._server_args_kwargs(kwargs)
self.generator = DiffGenerator.from_pretrained(
model_path=model_path,
pipeline_class_name=pipeline_class_name,
@@ -74,62 +202,132 @@ class SGLDiffusionGenerator:
)
return self.generator
def kill_generator(self):
"""Kill worker processes manually because generator shutdown cannot terminate them."""
current_pid = os.getpid()
worker_processes = []
for proc in psutil.process_iter(["pid", "name", "cmdline"]):
try:
# Look for sglang-diffusionWorker processes
if proc.info["cmdline"]:
cmdline = " ".join(proc.info["cmdline"])
if "sgl_diffusion::" in cmdline:
if proc.info["pid"] != current_pid:
worker_processes.append(proc)
except (psutil.NoSuchProcess, psutil.AccessDenied):
continue
@staticmethod
def _server_args_kwargs(kwargs: dict) -> dict:
"""Drop plugin-only / stale flags that ServerArgs no longer accepts."""
import dataclasses
if worker_processes:
logger.info(
f"Found {len(worker_processes)} worker processes to terminate..."
from sglang.multimodal_gen.runtime.server_args import ServerArgs
valid = {f.name for f in dataclasses.fields(ServerArgs)}
aliases = {"dp_degree": "dp_size", "cache_strategy": None, "model_type": None}
cleaned = {}
for key, value in kwargs.items():
dest = aliases.get(key, key)
if dest is None or dest not in valid:
continue
cleaned[dest] = value
return cleaned
def _worker_processes(self):
gen = self.generator
if gen is None:
return []
return list(getattr(gen, "local_scheduler_process", None) or [])
def _is_live(self) -> bool:
"""True when this owner still has a usable worker + scheduler client."""
if self.generator is None:
return False
try:
from sglang.multimodal_gen.runtime.scheduler_client import (
sync_scheduler_client,
)
for proc in worker_processes:
try:
logger.info(
f"Terminating worker process {proc.info['pid']}: {proc.info['name']}"
)
proc.terminate()
proc.wait(timeout=5)
except psutil.TimeoutExpired:
logger.warning(
f"Process {proc.info['pid']} did not terminate, forcing kill..."
)
try:
proc.kill()
proc.wait(timeout=2)
except (psutil.NoSuchProcess, psutil.TimeoutExpired):
pass
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
except ImportError:
return False
ctx = getattr(sync_scheduler_client, "context", None)
if ctx is None or getattr(ctx, "closed", False):
return False
workers = self._worker_processes()
if workers and not all(proc.is_alive() for proc in workers):
return False
return True
def _can_reuse(self, gather_options: dict) -> bool:
return (
self.last_options is not None
and self.last_options == gather_options
and self.generator is not None
and getattr(self, "_patcher", None) is not None
and self.executor is not None
and self._is_live()
)
def _owns_live(self, executor) -> bool:
return (
executor is not None
and self.generator is not None
and executor.generator is self.generator
and self._is_live()
)
def ensure_executor(self, executor) -> None:
"""Rebuild the worker if this cached executor no longer owns it."""
if self._owns_live(executor):
return
reload = getattr(executor, "_sgld_reload", None)
if not reload:
raise RuntimeError(
"SGLD worker is down and this executor has no reload spec"
)
logger.info(
"Rebuilding SGLD worker for %s (previous worker was replaced or died)",
reload.get("model_path"),
)
self.load_model(**reload)
executor.generator = self.generator
lora = getattr(executor, "_lora_input", None)
if lora and lora.get("lora_nickname"):
executor.set_lora(**lora)
def kill_generator(self):
"""Force-stop workers this owner started. Do not scan the process table."""
for proc in self._worker_processes():
if proc is None:
continue
try:
if not proc.is_alive():
continue
logger.info("Terminating SGLD worker pid=%s", getattr(proc, "pid", "?"))
proc.terminate()
proc.join(timeout=5)
if proc.is_alive():
proc.kill()
proc.join(timeout=2)
except Exception:
logger.warning(
"Failed to terminate SGLD worker %r", proc, exc_info=True
)
def close_generator(self):
"""Close and cleanup the generator and all associated resources."""
if self.generator is not None:
self.generator.shutdown()
try:
self.generator.shutdown()
except Exception:
logger.warning("DiffGenerator.shutdown failed", exc_info=True)
self.kill_generator()
# Clear other references
self.last_options = None
self.model_path = None
self.generator = None
self.executor = None
self.last_options = None
self.model_path = None
self.generator = None
self.executor = None
self._patcher = None
def get_comfyui_model(self, model_path: str, model_options: dict = None):
"""Get ComfyUI model from model path."""
if model_options is None:
model_options = {}
dtype = model_options.get("dtype", None)
# Allow loading unets from checkpoint files
sd = load_torch_file(model_path)
# Header-only read: H3 BF16 is ~66GB and we only need keys + shapes
# for detect_unet_config. SGLD loads the real weights later.
from comfy import model_detection, model_management
from comfy.utils import (
calculate_parameters,
state_dict_prefix_replace,
unet_to_diffusers,
)
sd = _load_state_dict_for_detection(model_path)
diffusion_model_prefix = model_detection.unet_prefix_from_state_dict(sd)
temp_sd = state_dict_prefix_replace(
sd, {diffusion_model_prefix: ""}, filter_keys=True
@@ -185,47 +383,78 @@ class SGLDiffusionGenerator:
self, model_path: str, model_options: dict = None, sgld_options: dict = None
):
"""Load model and return model patcher."""
gather_options = {
raw_sgld_options = dict(sgld_options) if sgld_options else {}
reload_kwargs = {
"model_path": model_path,
"model_options": dict(model_options) if model_options else {},
"sgld_options": raw_sgld_options,
}
sgld_options = dict(raw_sgld_options)
model_options = dict(reload_kwargs["model_options"])
plugin_flags = {}
if "enable_cache_dit" in sgld_options:
plugin_flags["enable_cache_dit"] = sgld_options.pop("enable_cache_dit")
set_model_type = sgld_options.pop("model_type", None)
override = (sgld_options.get("transformer_weights_path") or "").strip()
if override:
sgld_options["transformer_weights_path"] = override
elif _looks_like_gguf(model_path):
sgld_options["transformer_weights_path"] = model_path
else:
sgld_options.pop("transformer_weights_path", None)
detect_path = model_path
if _looks_like_gguf(model_path):
detect_path = _h3_detect_companion(sgld_options["transformer_weights_path"])
gather_options = {
"model_path": detect_path,
"model_options": model_options,
"sgld_options": sgld_options,
"set_model_type": set_model_type,
}
if (
self.last_options is not None
and self.last_options == gather_options
and self.generator is not None
):
return self.generator
else:
self.close_generator()
if self._can_reuse(gather_options):
self.executor.enable_cache_dit = plugin_flags.get("enable_cache_dit")
return self._patcher
self.close_generator()
self.last_options = gather_options
self.model_path = model_path
self.model_path = detect_path
comfyui_model, model_config, model_type = self.get_comfyui_model(
model_path, model_options
detect_path, model_options
)
if model_type is None or model_type not in self.pipeline_class_dict:
raise ValueError(f"Unsupported model type: {model_type}")
set_model_type = sgld_options.pop("model_type", None) if sgld_options else None
if set_model_type is not None and set_model_type in self.pipeline_class_dict:
model_type = set_model_type
pipeline_class_name = self.pipeline_class_dict[model_type]
self.generator = self.init_generator(
model_path, pipeline_class_name, sgld_options
detect_path, pipeline_class_name, sgld_options
)
executor_class = self.executor_class_dict[model_type]
self.executor = executor_class(
self.generator, model_path, comfyui_model, model_config
self.generator, detect_path, comfyui_model, model_config
)
self.executor.enable_cache_dit = plugin_flags.get("enable_cache_dit")
self.executor._sgld_reload = reload_kwargs
self.executor._ensure_runtime = self.ensure_executor
comfyui_model.diffusion_model = self.executor
from comfy import model_management
from comfy.patcher_extension import WrappersMP
from .model_patcher import SGLDModelPatcher
load_device = model_management.get_torch_device()
offload_device = model_management.unet_offload_device()
return SGLDModelPatcher(
self._patcher = SGLDModelPatcher(
comfyui_model, load_device, offload_device, model_type=model_type
)
self._patcher.add_wrapper_with_key(
WrappersMP.SAMPLER_SAMPLE,
"sgld_session",
self.executor.sampler_sample_wrapper,
)
return self._patcher
@@ -48,6 +48,17 @@ class SGLDModelPatcher(ModelPatcher):
n.backup = self.backup
n.object_patches_backup = self.object_patches_backup
n.lora_cache = copy.copy(self.lora_cache)
n.model_type = self.model_type
n.wrappers = {}
for wrapper_type, keyed in self.wrappers.items():
n.wrappers[wrapper_type] = {
key: wrappers.copy() for key, wrappers in keyed.items()
}
n.callbacks = {}
for call_type, keyed in self.callbacks.items():
n.callbacks[call_type] = {
key: callbacks.copy() for key, callbacks in keyed.items()
}
return n
def model_size(self):
@@ -3,15 +3,50 @@ ComfyUI SGLang Diffusion executors package.
Provides executor classes for different model types.
"""
from .adapter import ComfyUIModelAdapter, PackedForward, get_adapter_class
from .base import SGLDiffusionExecutor
from .flux import FluxExecutor
from .qwen_image import QwenImageEditExecutor, QwenImageExecutor
from .zimage import ZImageExecutor
from .flux import FluxAdapter, FluxExecutor
from .minimax_h3 import MiniMaxH3Adapter, MiniMaxH3Executor
from .zimage import ZImageAdapter, ZImageExecutor
# Qwen adapters import ComfyUI (`comfy.ldm.common_dit`). Keep that optional so
# unit tests and SGLD-only paths can load Flux / Z-Image without ComfyUI.
__all__ = [
"ComfyUIModelAdapter",
"PackedForward",
"SGLDiffusionExecutor",
"FluxAdapter",
"FluxExecutor",
"MiniMaxH3Adapter",
"MiniMaxH3Executor",
"ZImageAdapter",
"ZImageExecutor",
"QwenImageExecutor",
"QwenImageEditExecutor",
"get_adapter_class",
]
def __getattr__(name):
if name in {
"QwenImageExecutor",
"QwenImageEditExecutor",
"QwenImageAdapter",
"QwenImageEditAdapter",
}:
from .qwen_image import (
QwenImageAdapter,
QwenImageEditAdapter,
QwenImageEditExecutor,
QwenImageExecutor,
)
mapping = {
"QwenImageAdapter": QwenImageAdapter,
"QwenImageEditAdapter": QwenImageEditAdapter,
"QwenImageExecutor": QwenImageExecutor,
"QwenImageEditExecutor": QwenImageEditExecutor,
}
return mapping[name]
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
@@ -0,0 +1,87 @@
# SPDX-License-Identifier: Apache-2.0
"""Per-model adapters for the ComfyUI DiT-forward contract.
The shared executor owns request construction and the ZMQ round-trip.
Each adapter only translates between ComfyUI's ``apply_model`` tensors and
the fields SGLang's ``Req`` expects. Design the interface around H3 (nested
latents, structured payload), not Flux's three-tensor case.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
import torch
_ADAPTERS: dict[str, type[ComfyUIModelAdapter]] = {}
@dataclass
class PackedForward:
"""One ComfyUI sampler step, already translated into SGLang tensors."""
latents: torch.Tensor
timesteps: torch.Tensor
prompt_embeds: list[torch.Tensor]
height: int
width: int
guidance_scale: float = 1.0
prompt_seq_lens: list[list[int]] | None = None
pooled_embeds: list[torch.Tensor] | None = None
extra_req: dict[str, Any] = field(default_factory=dict)
unpack_ctx: dict[str, Any] = field(default_factory=dict)
class ComfyUIModelAdapter:
"""Model-specific pack / unpack / fill_req for one ComfyUI DiT family."""
model_types: tuple[str, ...] = ()
pipeline_class_name: str = ""
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
for model_type in cls.model_types:
_ADAPTERS[model_type] = cls
def pack(
self, x: torch.Tensor, timestep: torch.Tensor, context, **kwargs
) -> PackedForward:
raise NotImplementedError
def unpack(
self, noise_pred: torch.Tensor, packed: PackedForward, x: torch.Tensor
) -> torch.Tensor:
return noise_pred.to(x.device)
def fill_req(self, req, packed: PackedForward) -> None:
req.latents = packed.latents
req.timesteps = packed.timesteps
req.prompt_embeds = packed.prompt_embeds
req.raw_latent_shape = torch.tensor(packed.latents.shape, dtype=torch.long)
req.do_classifier_free_guidance = False
if packed.prompt_seq_lens is not None:
req.prompt_seq_lens = packed.prompt_seq_lens
if packed.pooled_embeds is not None:
req.pooled_embeds = packed.pooled_embeds
for key, value in packed.extra_req.items():
setattr(req, key, value)
def drop_cached_fields(self, packed: PackedForward) -> None:
packed.prompt_embeds = []
packed.prompt_seq_lens = None
packed.pooled_embeds = None
packed.extra_req.pop("image_latent", None)
def get_adapter_class(model_type: str) -> type[ComfyUIModelAdapter]:
if model_type not in _ADAPTERS:
raise ValueError(
f"Unsupported ComfyUI model type {model_type!r}. "
f"Registered: {sorted(_ADAPTERS)}"
)
return _ADAPTERS[model_type]
def registered_model_types() -> list[str]:
return sorted(_ADAPTERS)
@@ -2,11 +2,23 @@
Base executor class for SGLang Diffusion ComfyUI integration.
"""
import uuid
import torch
try:
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
except ImportError:
print(
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
)
class SGLDiffusionExecutor(torch.nn.Module):
"""Base executor class for SGLang Diffusion models in ComfyUI."""
"""Shared ComfyUI DiT-forward executor. Per-model logic lives on the adapter."""
adapter_cls = None
def __init__(self, generator, model_path, model, config):
super(SGLDiffusionExecutor, self).__init__()
@@ -16,6 +28,15 @@ class SGLDiffusionExecutor(torch.nn.Module):
self.dtype = config.unet_config["dtype"]
self.config = config
self.loras = []
self._lora_input = None
self._sgld_reload = None
self._ensure_runtime = None
if self.adapter_cls is None:
raise TypeError(f"{type(self).__name__} must set adapter_cls")
self.adapter = self.adapter_cls()
self.session_id = uuid.uuid4().hex
self._run_id = 0
self._sent_conds: set[tuple] = set()
@staticmethod
def should_suppress_logs(timestep):
@@ -26,7 +47,13 @@ class SGLDiffusionExecutor(torch.nn.Module):
def set_lora(self, lora_nickname=None, lora_path=None, strength=None, target=None):
"""Set LoRA adapter using SGLang Diffusion API."""
if len(lora_nickname) > 0:
self._lora_input = {
"lora_nickname": lora_nickname,
"lora_path": lora_path,
"strength": strength,
"target": target,
}
if lora_nickname and len(lora_nickname) > 0:
self.generator.set_lora(
lora_nickname=lora_nickname,
lora_path=lora_path,
@@ -34,23 +61,87 @@ class SGLDiffusionExecutor(torch.nn.Module):
target=target,
)
def _unpack_latents(self, latents, height, width, channels):
"""Unpack latents from packed format to standard format."""
batch_size = latents.shape[0]
latents = latents.view(batch_size, height // 2, width // 2, channels, 2, 2)
latents = latents.permute(0, 3, 1, 4, 2, 5)
latents = latents.reshape(batch_size, channels, height, width)
def begin_sampler_run(self) -> None:
"""One ComfyUI ``sampler.sample()`` invocation is one cache lifetime."""
self._run_id += 1
self._sent_conds = set()
return latents
def end_sampler_run(self) -> None:
"""Run cache is evicted on the next bind of a newer id for this executor."""
def _pack_latents(self, latents):
"""Pack latents from standard format to packed format."""
batch_size, num_channels_latents, height, width = latents.shape
latents = latents.view(
batch_size, num_channels_latents, height // 2, 2, width // 2, 2
def sampler_sample_wrapper(self, executor, *args, **kwargs):
self.begin_sampler_run()
try:
return executor(*args, **kwargs)
finally:
self.end_sampler_run()
def comfyui_session_id(self) -> str:
return f"{self.session_id}:{self._run_id}"
def _cond_key(self, packed) -> tuple | None:
embeds = packed.prompt_embeds
if not embeds:
return None
tensor = embeds[0]
if not torch.is_tensor(tensor) or tensor.numel() == 0:
return None
flat = tensor.reshape(-1)
return (
tuple(int(dim) for dim in tensor.shape),
float(flat[0].item()),
float(flat[-1].item()),
)
latents = latents.permute(0, 2, 4, 1, 3, 5)
latents = latents.reshape(
batch_size, (height // 2) * (width // 2), num_channels_latents * 4
def _mark_and_maybe_drop(self, packed) -> None:
key = self._cond_key(packed)
if key is not None:
packed.extra_req["comfyui_cond_key"] = repr(key)
if key in self._sent_conds:
self.adapter.drop_cached_fields(packed)
else:
self._sent_conds.add(key)
def _sampling_params_kwargs(self, packed, timestep) -> dict:
return {
"prompt": " ",
"guidance_scale": packed.guidance_scale,
"height": packed.height,
"width": packed.width,
"num_frames": 1,
"num_inference_steps": 1,
"save_output": False,
"suppress_logs": self.should_suppress_logs(timestep),
}
def _execute_packed(self, packed, x, timestep):
ensure = getattr(self, "_ensure_runtime", None)
if ensure is not None:
ensure(self)
self._mark_and_maybe_drop(packed)
sampling_params = SamplingParams.from_user_sampling_params_args(
self.model_path,
server_args=self.generator.server_args,
**self._sampling_params_kwargs(packed, timestep),
)
return latents
req = prepare_request(
server_args=self.generator.server_args,
sampling_params=sampling_params,
)
self.adapter.fill_req(req, packed)
extra = dict(req.extra or {})
extra["comfyui_session_id"] = self.comfyui_session_id()
for key in ("comfyui_cond_key", "comfyui_cache_fp"):
value = packed.extra_req.get(key)
if value is not None:
extra[key] = value
req.extra = extra
req.generator = [
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
]
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
return self.adapter.unpack(output_batch.noise_pred, packed, x)
def forward(self, x, timestep, context, **kwargs):
packed = self.adapter.pack(x, timestep, context, **kwargs)
return self._execute_packed(packed, x, timestep)
@@ -1,69 +1,69 @@
"""
Flux executor for SGLang Diffusion ComfyUI integration.
"""
"""Flux adapter for the ComfyUI DiT-forward contract."""
import torch
try:
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
except ImportError:
print(
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
)
from .adapter import ComfyUIModelAdapter, PackedForward
from .base import SGLDiffusionExecutor
def _flux_guidance_scale(guidance) -> float:
if guidance is None:
return 3.5
if torch.is_tensor(guidance):
return float(guidance.detach().reshape(-1)[0].item())
return float(guidance)
class FluxAdapter(ComfyUIModelAdapter):
model_types = ("flux",)
pipeline_class_name = "FluxPipeline"
def pack(
self, x, timestep, context, y=None, guidance=None, **kwargs
) -> PackedForward:
packed = self._pack_latents(x)
t5_seq = int(context.shape[-2]) if context.ndim >= 2 else int(context.shape[0])
clip_batch = int(y.shape[0]) if y is not None else 1
return PackedForward(
latents=packed,
timesteps=timestep * 1000.0,
prompt_embeds=[y, context],
prompt_seq_lens=[[clip_batch], [t5_seq]],
pooled_embeds=[y],
height=x.shape[-2] * 8,
width=x.shape[-1] * 8,
guidance_scale=_flux_guidance_scale(guidance),
unpack_ctx={
"height": x.shape[-2],
"width": x.shape[-1],
"channels": x.shape[1],
},
)
def unpack(self, noise_pred, packed, x):
ctx = packed.unpack_ctx
return self._unpack_latents(
noise_pred, ctx["height"], ctx["width"], ctx["channels"]
).to(x.device)
@staticmethod
def _unpack_latents(latents, height, width, channels):
batch_size = latents.shape[0]
latents = latents.view(batch_size, height // 2, width // 2, channels, 2, 2)
latents = latents.permute(0, 3, 1, 4, 2, 5)
return latents.reshape(batch_size, channels, height, width)
@staticmethod
def _pack_latents(latents):
batch_size, num_channels_latents, height, width = latents.shape
latents = latents.view(
batch_size, num_channels_latents, height // 2, 2, width // 2, 2
)
latents = latents.permute(0, 2, 4, 1, 3, 5)
return latents.reshape(
batch_size, (height // 2) * (width // 2), num_channels_latents * 4
)
class FluxExecutor(SGLDiffusionExecutor):
"""Executor for Flux models in ComfyUI."""
def __init__(self, generator, model_path, model, config):
super().__init__(generator, model_path, model, config)
def forward(self, x, timestep, context, y=None, guidance=None, **kwargs):
"""Forward pass for Flux model."""
hidden_states = self._pack_latents(x)
timesteps = timestep * 1000.0
encoder_hidden_states = context
pooled_projections = y
guidance = guidance * 1000.0
B, C, H, W = x.shape
height = H * 8
width = W * 8
# Create SamplingParams
sampling_params = SamplingParams.from_user_sampling_params_args(
self.model_path,
server_args=self.generator.server_args,
prompt=" ",
guidance_scale=3.5, # Flux typically uses embedded_cfg_scale=3.5
height=height,
width=width,
num_frames=1,
num_inference_steps=1,
save_output=False,
suppress_logs=self.should_suppress_logs(timestep),
)
# Prepare request (converts SamplingParams to Req)
req = prepare_request(
server_args=self.generator.server_args,
sampling_params=sampling_params,
)
req.latents = hidden_states # Set as [B, S, D] format directly
req.timesteps = timesteps # ComfyUI's timesteps parameter
req.prompt_embeds = [pooled_projections, encoder_hidden_states] # [CLIP, T5]
req.raw_latent_shape = torch.tensor(hidden_states.shape, dtype=torch.long)
# Set pooled_projections (required by Flux)
req.pooled_embeds = [pooled_projections] # List format as per Req definition
req.do_classifier_free_guidance = False
req.generator = [
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
]
# Send request to scheduler
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
noise_pred = output_batch.noise_pred
return self._unpack_latents(noise_pred, H, W, C).to(x.device)
adapter_cls = FluxAdapter
@@ -0,0 +1,230 @@
# SPDX-License-Identifier: Apache-2.0
"""MiniMax-H3 adapter for the ComfyUI DiT-forward contract."""
from __future__ import annotations
from dataclasses import fields
from typing import Any
import torch
from sglang.multimodal_gen.configs.sample.minimax_h3 import MiniMaxH3SamplingParams
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import (
DEFAULT_SIGMA_SHIFT_AUDIO,
DEFAULT_SIGMA_SHIFT_VIDEO,
serialize_comfyui_layout,
time_shift_sigma,
time_shift_slope,
)
from .adapter import ComfyUIModelAdapter, PackedForward
from .base import SGLDiffusionExecutor
# H3 pins loop/CFG fields with init=False; ComfyUI owns those.
_H3_PINNED_SAMPLING_FIELDS = frozenset(
f.name for f in fields(MiniMaxH3SamplingParams) if not f.init
)
def drop_h3_pinned_sampling_fields(kwargs: dict[str, Any]) -> dict[str, Any]:
for name in _H3_PINNED_SAMPLING_FIELDS:
kwargs.pop(name, None)
return kwargs
_WORKER_TRANSFORMER_OPTION_KEYS = (
"minimax_h3_sigma_shift_video",
"minimax_h3_sigma_shift_audio",
"sample_sigmas",
)
def worker_transformer_options(opts: dict[str, Any] | None) -> dict[str, Any]:
"""Keep only pickle-safe H3 keys; drop ComfyUI model_sampling etc."""
src = opts or {}
return {key: src[key] for key in _WORKER_TRANSFORMER_OPTION_KEYS if key in src}
def _split_av(x) -> tuple[torch.Tensor, torch.Tensor]:
if isinstance(x, (list, tuple)) and len(x) >= 2:
return x[0], x[1]
unbind = getattr(x, "unbind", None)
if callable(unbind):
parts = unbind()
if len(parts) >= 2:
return parts[0], parts[1]
raise TypeError(
f"MiniMax H3 ComfyUI forward expects x to be [video, audio], got {type(x)!r}"
)
def _as_context(context: torch.Tensor) -> torch.Tensor:
if context.ndim == 3:
return context[0]
return context
def _cache_fingerprint(video, sample_sigmas, serialized_layout) -> dict[str, Any]:
spatial = tuple(int(dim) for dim in video.shape[-3:])
sigmas_fp = None
if sample_sigmas is not None:
values = (
sample_sigmas.reshape(-1)
if torch.is_tensor(sample_sigmas)
else sample_sigmas
)
sigmas_fp = (float(values[0]), float(values[-1]), int(len(values)))
layout_sig = None
if isinstance(serialized_layout, dict):
layout_sig = serialized_layout.get("signature")
return {"spatial": spatial, "sigmas": sigmas_fp, "layout": layout_sig}
class MiniMaxH3Adapter(ComfyUIModelAdapter):
model_types = ("minimax_h3",)
pipeline_class_name = "MiniMaxH3Pipeline"
def pack(self, x, timestep, context, **kwargs) -> PackedForward:
video, audio_src = _split_av(x)
# PackedLayout is a ComfyUI class; the SGLD worker cannot unpickle it.
payload = dict(kwargs.get("minimax_payload") or {})
layout = payload.pop("layout", None)
transformer_options = kwargs.get("transformer_options") or {}
scale = float(payload.get("audio_scale", 1.0))
shift_v = float(
transformer_options.get(
"minimax_h3_sigma_shift_video", DEFAULT_SIGMA_SHIFT_VIDEO
)
)
shift_a = float(
transformer_options.get(
"minimax_h3_sigma_shift_audio", DEFAULT_SIGMA_SHIFT_AUDIO
)
)
sigma_v = (timestep.reshape(-1)[0] / 1000.0).to(torch.float32).clamp(min=1e-6)
sigma_a = time_shift_sigma(sigma_v, shift_v, shift_a)
carry = (sigma_a / sigma_v).to(audio_src.dtype)
audio = audio_src * carry if scale != 1.0 else audio_src
text = _as_context(context)
sample_sigmas = transformer_options.get("sample_sigmas")
extra_req: dict[str, Any] = {
"audio_latents": audio,
"h3_payload": payload,
"h3_transformer_options": worker_transformer_options(transformer_options),
"h3_sample_sigmas": sample_sigmas,
"h3_context": text,
}
serialized = serialize_comfyui_layout(layout)
if serialized is not None:
extra_req["h3_layout"] = serialized
for name in ("denoise_mask", "audio_denoise_mask"):
mask = kwargs.get(name)
if mask is not None:
extra_req[f"h3_{name}"] = mask
if sample_sigmas is not None:
extra_req["sigmas"] = sample_sigmas
extra_req["comfyui_cache_fp"] = _cache_fingerprint(
video, sample_sigmas, serialized
)
return PackedForward(
latents=video,
timesteps=timestep,
prompt_embeds=[text],
prompt_seq_lens=[[int(text.shape[0])]],
height=int(video.shape[-2]),
width=int(video.shape[-1]),
extra_req=extra_req,
unpack_ctx={
"audio_scale": scale,
"audio_src": audio_src,
"carry": carry,
"sigma_v": sigma_v,
"sigma_a": sigma_a,
"shift_v": shift_v,
"shift_a": shift_a,
},
)
def unpack(self, noise_pred, packed, x):
video_x, audio_x = _split_av(x)
if isinstance(noise_pred, (list, tuple)):
v_video, v_audio = noise_pred[0], noise_pred[1]
else:
raise TypeError(
"MiniMax H3 unpack expects [video, audio] noise_pred, "
f"got {type(noise_pred)!r}"
)
# Match ComfyUI MiniMaxH3._forward: negated velocities, and audio
# scaled by d(sigma_a)/d(sigma_v) so the sampler's sigma_v ODE is
# the audio stream's true ODE on its shifted schedule.
v_video = (-v_video).to(device=video_x.device, dtype=video_x.dtype)
v_audio = (-v_audio).to(device=audio_x.device, dtype=audio_x.dtype)
ctx = packed.unpack_ctx
scale = float(ctx.get("audio_scale", 1.0))
if scale != 1.0:
audio_src = ctx["audio_src"]
carry = ctx["carry"]
sigma_a = ctx["sigma_a"]
v_audio = (1.0 - scale) * (audio_src * carry) + (
1.0 + (scale - 1.0) * sigma_a
).to(v_audio.dtype) * v_audio
slope_a = time_shift_slope(
ctx["sigma_v"],
ctx.get("shift_v", DEFAULT_SIGMA_SHIFT_VIDEO),
ctx.get("shift_a", DEFAULT_SIGMA_SHIFT_AUDIO),
).to(device=v_audio.device, dtype=v_audio.dtype)
v_audio = slope_a * v_audio
return [v_video, v_audio]
def fill_req(self, req, packed: PackedForward) -> None:
super().fill_req(req, packed)
extra = dict(req.extra or {})
for key in (
"h3_payload",
"h3_transformer_options",
"h3_sample_sigmas",
"h3_context",
"h3_layout",
"h3_denoise_mask",
"h3_audio_denoise_mask",
"comfyui_cache_fp",
"comfyui_cond_key",
):
value = packed.extra_req.get(key)
if value is not None:
extra[key] = value
req.extra = extra
def drop_cached_fields(self, packed: PackedForward) -> None:
super().drop_cached_fields(packed)
packed.extra_req.pop("h3_payload", None)
packed.extra_req.pop("h3_transformer_options", None)
packed.extra_req.pop("h3_sample_sigmas", None)
packed.extra_req.pop("h3_context", None)
packed.extra_req.pop("h3_layout", None)
packed.extra_req.pop("h3_denoise_mask", None)
packed.extra_req.pop("h3_audio_denoise_mask", None)
packed.extra_req.pop("sigmas", None)
# Keep comfyui_cache_fp / comfyui_cond_key; later steps still send them.
class MiniMaxH3Executor(SGLDiffusionExecutor):
adapter_cls = MiniMaxH3Adapter
# ComfyUI MiniMaxH3.extra_conds reads this when a denoise_mask is present.
patch_size = (1, 2, 2)
def preprocess_text_embeds(self, text_states):
"""ComfyUI extra_conds runs this once per sample.
Token refinement happens on the SGLD worker (the DiT lives there).
Identity-pass here so PackedLayout still sees the correct token length.
"""
return text_states
def _sampling_params_kwargs(self, packed, timestep) -> dict:
kwargs = super()._sampling_params_kwargs(packed, timestep)
drop_h3_pinned_sampling_fields(kwargs)
enable_cache_dit = getattr(self, "enable_cache_dit", None)
if enable_cache_dit is not None:
kwargs["enable_cache_dit"] = bool(enable_cache_dit)
return kwargs
@@ -1,31 +1,40 @@
"""
QwenImage executor for SGLang Diffusion ComfyUI integration.
"""
import torch
try:
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
except ImportError:
print(
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
)
"""Qwen-Image adapters for the ComfyUI DiT-forward contract."""
import comfy.ldm.common_dit
from .adapter import ComfyUIModelAdapter, PackedForward
from .base import SGLDiffusionExecutor
class QwenImageExecutor(SGLDiffusionExecutor):
"""Executor for QwenImage models in ComfyUI."""
class QwenImageAdapter(ComfyUIModelAdapter):
model_types = ("qwen_image",)
pipeline_class_name = "QwenImagePipeline"
patch_size = 2
def __init__(self, generator, model_path, model, config):
super().__init__(generator, model_path, model, config)
self.patch_size = 2
def pack(self, x, timestep, context, **kwargs) -> PackedForward:
latents, orig_shape = self._pack_latents(x)
seq = int(context.shape[-2]) if context.ndim >= 2 else int(context.shape[0])
return PackedForward(
latents=latents,
timesteps=timestep * 1000.0,
prompt_embeds=[context],
prompt_seq_lens=[[seq]],
height=orig_shape[-2] * 8,
width=orig_shape[-1] * 8,
unpack_ctx={
"num_embeds": latents.shape[1],
"orig_shape": orig_shape,
"x": x,
},
)
def unpack(self, noise_pred, packed, x):
ctx = packed.unpack_ctx
return self._unpack_latents(
noise_pred, ctx["num_embeds"], ctx["orig_shape"], ctx["x"]
)
def _pack_latents(self, x):
"""Process hidden states for QwenImage model."""
latents = comfy.ldm.common_dit.pad_to_patch_size(
x, (1, self.patch_size, self.patch_size)
)
@@ -47,8 +56,8 @@ class QwenImageExecutor(SGLDiffusionExecutor):
)
return latents, orig_shape
def _unpack_latents(self, latents, num_embeds, orig_shape, x):
"""Unpack hidden states from packed format to standard format."""
@staticmethod
def _unpack_latents(latents, num_embeds, orig_shape, x):
latents = latents[:, :num_embeds].view(
orig_shape[0],
orig_shape[-3],
@@ -59,57 +68,14 @@ class QwenImageExecutor(SGLDiffusionExecutor):
2,
)
latents = latents.permute(0, 4, 1, 2, 5, 3, 6)
latents = latents.reshape(orig_shape)[:, :, :, : x.shape[-2], : x.shape[-1]]
return latents
def forward(self, x, timestep, context, **kwargs):
"""Forward pass for QwenImage model."""
latents, orig_shape = self._pack_latents(x)
num_embeds = latents.shape[1]
height = orig_shape[-2] * 8
width = orig_shape[-1] * 8
sampling_params = SamplingParams.from_user_sampling_params_args(
self.model_path,
server_args=self.generator.server_args,
prompt=" ",
guidance_scale=1.0,
height=height,
width=width,
num_frames=1,
num_inference_steps=1,
save_output=False,
suppress_logs=self.should_suppress_logs(timestep),
)
# Prepare request (converts SamplingParams to Req)
req = prepare_request(
server_args=self.generator.server_args,
sampling_params=sampling_params,
)
# Set ComfyUI-specific inputs directly on the Req object
req.latents = latents
req.timesteps = timestep * 1000.0
req.prompt_embeds = [context]
req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long)
req.do_classifier_free_guidance = False
req.generator = [
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
]
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
noise_pred = output_batch.noise_pred
return self._unpack_latents(noise_pred, num_embeds, orig_shape, x)
return latents.reshape(orig_shape)[:, :, :, : x.shape[-2], : x.shape[-1]]
class QwenImageEditExecutor(QwenImageExecutor):
"""Executor for QwenImageEdit models in ComfyUI."""
class QwenImageEditAdapter(QwenImageAdapter):
model_types = ("qwen_image_edit",)
pipeline_class_name = "QwenImageEditPlusPipeline"
def __init__(self, generator, model_path, model, config):
super().__init__(generator, model_path, model, config)
def forward(
def pack(
self,
x,
timestep,
@@ -117,56 +83,22 @@ class QwenImageEditExecutor(QwenImageExecutor):
attention_mask=None,
ref_latents=None,
additional_t_cond=None,
transformer_options={},
transformer_options=None,
**kwargs,
):
"""Forward pass for QwenImageEdit model."""
latents, orig_shape = self._pack_latents(x)
num_embeds = latents.shape[1]
height = orig_shape[-2] * 8
width = orig_shape[-1] * 8
) -> PackedForward:
packed = super().pack(x, timestep, context, **kwargs)
if ref_latents:
pack_ref, orig_ref_shape = self._pack_latents(ref_latents[0])
packed.extra_req["image_latent"] = pack_ref
packed.extra_req["vae_image_sizes"] = [
(orig_ref_shape[-1], orig_ref_shape[-2])
]
return packed
# Prepare vae_image_sizes for the condition image (ref_latents)
vae_image_sizes = []
pack_ref_latents = None
# TODO: sgld now don't support multiple condition images, so we only support one condition image for now.
if ref_latents is not None and len(ref_latents) > 0:
pack_ref_latents, orig_ref_shape = self._pack_latents(ref_latents[0])
vae_image_sizes = [(orig_ref_shape[-1], orig_ref_shape[-2])]
class QwenImageExecutor(SGLDiffusionExecutor):
adapter_cls = QwenImageAdapter
sampling_params = SamplingParams.from_user_sampling_params_args(
self.model_path,
server_args=self.generator.server_args,
prompt=" ",
guidance_scale=1.0,
image_path="",
height=height,
width=width,
num_frames=1,
num_inference_steps=1,
save_output=False,
suppress_logs=self.should_suppress_logs(timestep),
)
# Prepare request (converts SamplingParams to Req)
req = prepare_request(
server_args=self.generator.server_args,
sampling_params=sampling_params,
)
# Set ComfyUI-specific inputs directly on the Req object
req.latents = latents
req.image_latent = pack_ref_latents
req.timesteps = timestep * 1000.0
req.vae_image_sizes = vae_image_sizes
req.prompt_embeds = [context]
req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long)
req.do_classifier_free_guidance = False
req.generator = [
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
]
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
noise_pred = output_batch.noise_pred
return self._unpack_latents(noise_pred, num_embeds, orig_shape, x)
class QwenImageEditExecutor(SGLDiffusionExecutor):
adapter_cls = QwenImageEditAdapter
@@ -1,64 +1,30 @@
"""
ZImage executor for SGLang Diffusion ComfyUI integration.
"""
import torch
try:
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
except ImportError:
print(
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
)
"""Z-Image (lumina2) adapter for the ComfyUI DiT-forward contract."""
from .adapter import ComfyUIModelAdapter, PackedForward
from .base import SGLDiffusionExecutor
class ZImageExecutor(SGLDiffusionExecutor):
"""Executor for ZImage models in ComfyUI."""
class ZImageAdapter(ComfyUIModelAdapter):
model_types = ("lumina2",)
pipeline_class_name = "ZImagePipeline"
def __init__(self, generator, model_path, model, config):
super().__init__(generator, model_path, model, config)
def forward(self, x, timesteps, context, **kwargs):
"""Forward pass for ZImage model."""
B, C, H, W = x.shape
height = H * 8
width = W * 8
sampling_params = SamplingParams.from_user_sampling_params_args(
self.model_path,
server_args=self.generator.server_args,
prompt=" ",
guidance_scale=1.0,
height=height,
width=width,
num_frames=1, # For images
num_inference_steps=1, # Single step for ComfyUI
save_output=False,
suppress_logs=self.should_suppress_logs(timesteps),
)
# Prepare request (converts SamplingParams to Req)
req = prepare_request(
server_args=self.generator.server_args,
sampling_params=sampling_params,
)
latents = x.unsqueeze(2)
def pack(self, x, timestep, context, **kwargs) -> PackedForward:
context = context.squeeze(0)
# Set ComfyUI-specific inputs directly on the Req object
req.latents = latents # ComfyUI's x parameter
req.timesteps = timesteps * 1000.0 # ComfyUI's timesteps parameter
req.prompt_embeds = [
context
] # ComfyUI's context parameter (must be List[Tensor])
req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long)
req.do_classifier_free_guidance = False
req.generator = [
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
]
return PackedForward(
latents=x.unsqueeze(2),
timesteps=timestep * 1000.0,
prompt_embeds=[context],
prompt_seq_lens=[[int(context.shape[0])]],
height=x.shape[-2] * 8,
width=x.shape[-1] * 8,
)
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
noise_pred = output_batch.noise_pred
def unpack(self, noise_pred, packed, x):
# SGLD returns 5D [B, C, T, H, W]; ComfyUI samples 4D [B, C, H, W].
if noise_pred.ndim == 5:
noise_pred = noise_pred.squeeze(2)
return noise_pred.to(x.device)
return noise_pred.permute(1, 0, 2, 3).to(x.device)
class ZImageExecutor(SGLDiffusionExecutor):
adapter_cls = ZImageAdapter
@@ -10,6 +10,16 @@ import folder_paths
import torch
from .core import SGLDiffusionGenerator, SGLDiffusionServerAPI
def _enable_gguf_in_diffusion_models() -> None:
"""Let SGLDUNETLoader list ``.gguf`` DiTs. Quantization lives in the file."""
entry = folder_paths.folder_names_and_paths.get("diffusion_models")
if entry is not None and len(entry) >= 2 and isinstance(entry[1], set):
entry[1].add(".gguf")
_enable_gguf_in_diffusion_models()
from .utils import (
convert_b64_to_tensor_image,
convert_video_to_comfy_video,
@@ -25,7 +35,14 @@ class SGLDOptions:
"required": {},
"optional": {
"model_type": (
["auto-detect", "qwen_image", "qwen_image_edit", "flux", "lumina2"],
[
"auto-detect",
"qwen_image",
"qwen_image_edit",
"flux",
"lumina2",
"minimax_h3",
],
{"default": "auto-detect"},
),
"enable_torch_compile": (
@@ -61,6 +78,25 @@ class SGLDOptions:
"STRING",
{"default": ""},
),
"dit_layerwise_offload": (
"BOOLEAN",
{"default": False},
),
"enable_cache_dit": (
"BOOLEAN",
{"default": False},
),
"quantization": (
"STRING",
{"default": ""},
),
"transformer_weights_path": (
"STRING",
{
"default": "",
"multiline": False,
},
),
},
}
@@ -82,6 +118,10 @@ class SGLDOptions:
dp_degree: int = 1,
enable_cfg_parallel: bool = False,
attention_backend: str = "",
dit_layerwise_offload: bool = False,
enable_cache_dit: bool = False,
quantization: str = "",
transformer_weights_path: str = "",
):
"""
Build a dictionary of SGLang Diffusion runtime options.
@@ -89,7 +129,12 @@ class SGLDOptions:
# Convert -1 to None for optional parameters (matching ServerArgs defaults)
ulysses_degree = None if ulysses_degree == -1 else ulysses_degree
ring_degree = None if ring_degree == -1 else ring_degree
tp_size = None if tp_size == -1 else tp_size
sp_degree = None if sp_degree == -1 else sp_degree
attention_backend = None if attention_backend == "" else attention_backend
# dp_degree is a leftover alias; ServerArgs only has dp_size.
if dp_degree not in (None, 1) and dp_size in (None, 1):
dp_size = dp_degree
options = {
"model_type": model_type,
@@ -100,10 +145,20 @@ class SGLDOptions:
"ulysses_degree": ulysses_degree,
"ring_degree": ring_degree,
"dp_size": dp_size,
"dp_degree": dp_degree,
"enable_cfg_parallel": enable_cfg_parallel,
"attention_backend": attention_backend,
"dit_layerwise_offload": dit_layerwise_offload,
}
if enable_cache_dit:
options["enable_cache_dit"] = True
quantization = (quantization or "").strip()
if quantization:
options["quantization"] = quantization
transformer_weights_path = (transformer_weights_path or "").strip()
if transformer_weights_path:
# Same selector as `sglang serve --transformer-weights-path`:
# local .gguf, owner/repo/path/file.gguf, or owner/repo:QUANT.
options["transformer_weights_path"] = transformer_weights_path
# Strip None to keep payload clean
options = {k: v for k, v in options.items() if v is not None}
@@ -165,7 +220,7 @@ class SGLDLoraLoader:
class SGLDUNETLoader:
def __init__(self):
self.generator = SGLDiffusionGenerator()
self.generator = SGLDiffusionGenerator.shared()
@classmethod
def INPUT_TYPES(s):
@@ -1,13 +1,16 @@
# ComfyUI SGLDiffusion Pipeline Tests
This directory contains tests for each ComfyUI pipeline integration.
These e2e tests start the **native** Flux / Qwen-Image / Z-Image pipeline with
`comfyui_mode=True` and feed pre-encoded latents, timesteps, and embeddings the
way the plugin adapter would. Adapter, session, and CUDA IPC unit tests live
under `python/sglang/multimodal_gen/test/unit/`.
## Test Files
- `test_zimage_pipeline.py` - Tests for ComfyUIZImagePipeline
- `test_flux_pipeline.py` - Tests for ComfyUIFluxPipeline
- `test_qwen_image_pipeline.py` - Tests for ComfyUIQwenImagePipeline
- `test_qwen_image_edit_pipeline.py` - Tests for ComfyUIQwenImageEditPipeline (I2I/edit mode)
- `test_zimage_pipeline.py` - Z-Image native pipeline under `--comfyui-mode`
- `test_flux_pipeline.py` - Flux native pipeline under `--comfyui-mode`
- `test_qwen_image_pipeline.py` - Qwen-Image native pipeline under `--comfyui-mode`
- `test_qwen_image_edit_pipeline.py` - Qwen-Image-Edit native pipeline (I2I/edit; experimental)
## Running Tests
@@ -1,4 +1,4 @@
"""Test for ComfyUIFluxPipeline with pass-through scheduler."""
"""Test for FluxPipeline with pass-through scheduler."""
import os
import sys
@@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
def test_comfyui_flux_pipeline_direct() -> None:
"""Test ComfyUIFluxPipeline with custom inputs."""
"""Test FluxPipeline with custom inputs."""
model_path = os.environ.get(
"SGLANG_TEST_FLUX_MODEL_PATH",
"black-forest-labs/FLUX.1-dev", # Supports both safetensors file and diffusers format
@@ -20,7 +20,8 @@ def test_comfyui_flux_pipeline_direct() -> None:
generator = DiffGenerator.from_pretrained(
model_path=model_path,
pipeline_class_name="ComfyUIFluxPipeline",
model_id="FLUX.1-dev",
pipeline_class_name="FluxPipeline",
num_gpus=2,
comfyui_mode=True,
)
@@ -68,6 +69,7 @@ def test_comfyui_flux_pipeline_direct() -> None:
width=width,
num_frames=1,
num_inference_steps=1,
guidance_scale=1.0,
save_output=True,
return_trajectory_latents=True,
)
@@ -83,6 +85,10 @@ def test_comfyui_flux_pipeline_direct() -> None:
clip_dim = 768
req.prompt_embeds = [pooled_projections, encoder_hidden_states]
req.prompt_seq_lens = [
[int(pooled_projections.shape[0])],
[encoder_seq_len],
]
if req.guidance_scale > 1.0:
dummy_neg_clip_embedding = torch.zeros(
@@ -103,11 +109,15 @@ def test_comfyui_flux_pipeline_direct() -> None:
dummy_neg_clip_embedding,
negative_encoder_hidden_states,
]
req.negative_prompt_seq_lens = [
[int(dummy_neg_clip_embedding.shape[0])],
[encoder_seq_len],
]
else:
req.negative_prompt_embeds = None
req.pooled_embeds = [pooled_projections]
req.neg_pooled_embeds = []
req.neg_pooled_embeds = [torch.zeros_like(pooled_projections)]
if (
req.guidance_scale > 1.0
@@ -1,4 +1,4 @@
"""Test for ComfyUIQwenImageEditPipeline with pass-through scheduler (I2I/edit mode)."""
"""Test for QwenImageEditPlusPipeline with pass-through scheduler (I2I/edit mode)."""
import os
import sys
@@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
def test_comfyui_qwen_image_edit_pipeline_direct() -> None:
"""Test ComfyUIQwenImageEditPipeline with edit mode (I2I) and custom inputs."""
"""Test QwenImageEditPlusPipeline with edit mode (I2I) and custom inputs."""
model_path = os.environ.get(
"SGLANG_TEST_QWEN_IMAGE_EDIT_MODEL_PATH",
"Qwen/Qwen-Image-Edit-2511", # Supports both safetensors file and diffusers format
@@ -20,7 +20,7 @@ def test_comfyui_qwen_image_edit_pipeline_direct() -> None:
generator = DiffGenerator.from_pretrained(
model_path=model_path,
pipeline_class_name="ComfyUIQwenImageEditPipeline",
pipeline_class_name="QwenImageEditPlusPipeline",
num_gpus=1,
comfyui_mode=True,
dit_layerwise_offload=False,
@@ -1,4 +1,4 @@
"""Test for ComfyUIQwenImagePipeline with pass-through scheduler."""
"""Test for QwenImagePipeline with pass-through scheduler."""
import os
import sys
@@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
def test_comfyui_qwen_image_pipeline_direct() -> None:
"""Test ComfyUIQwenImagePipeline with custom inputs."""
"""Test QwenImagePipeline with custom inputs."""
model_path = os.environ.get(
"SGLANG_TEST_QWEN_IMAGE_MODEL_PATH",
"Qwen/Qwen-Image", # Supports both safetensors file and diffusers format
@@ -20,7 +20,7 @@ def test_comfyui_qwen_image_pipeline_direct() -> None:
generator = DiffGenerator.from_pretrained(
model_path=model_path,
pipeline_class_name="ComfyUIQwenImagePipeline",
pipeline_class_name="QwenImagePipeline",
num_gpus=2,
comfyui_mode=True,
dit_layerwise_offload=False,
@@ -1,4 +1,4 @@
"""Test for ComfyUIZImagePipeline with pass-through scheduler."""
"""Test for ZImagePipeline with pass-through scheduler."""
import os
import sys
@@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
def test_comfyui_zimage_pipeline_direct() -> None:
"""Test ComfyUIZImagePipeline with custom inputs."""
"""Test ZImagePipeline with custom inputs."""
model_path = os.environ.get(
"SGLANG_TEST_ZIMAGE_MODEL_PATH",
"Tongyi-MAI/Z-Image-Turbo", # Supports both safetensors file and diffusers format
@@ -20,7 +20,7 @@ def test_comfyui_zimage_pipeline_direct() -> None:
generator = DiffGenerator.from_pretrained(
model_path=model_path,
pipeline_class_name="ComfyUIZImagePipeline",
pipeline_class_name="ZImagePipeline",
num_gpus=1,
sp_degree=1,
comfyui_mode=True,
@@ -77,6 +77,7 @@ def test_comfyui_zimage_pipeline_direct() -> None:
req.latents = latents
req.timesteps = timesteps
req.prompt_embeds = [context]
req.prompt_seq_lens = [[context_seq_len]]
req.negative_prompt_embeds = None
req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long)
@@ -0,0 +1,136 @@
{
"6": {
"class_type": "SGLDUNETLoader",
"inputs": {
"unet_name": "minimax_h3_ref2va_bf16.safetensors",
"weight_dtype": "default",
"sgld_options": ["52", 0]
}
},
"52": {
"class_type": "SGLDOptions",
"inputs": {
"model_type": "minimax_h3",
"enable_torch_compile": false,
"num_gpus": 1,
"tp_size": -1,
"sp_degree": -1,
"ulysses_degree": -1,
"ring_degree": -1,
"dp_size": 1,
"dp_degree": 1,
"enable_cfg_parallel": false,
"attention_backend": ""
}
},
"13": {
"class_type": "CLIPLoader",
"inputs": {
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
"type": "minimax",
"device": "default"
}
},
"11": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
}
},
"24": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_audio_vae_fp32.safetensors"
}
},
"137": {
"class_type": "LoadImage",
"inputs": {
"image": "sglang_girl.jpg"
}
},
"136": {
"class_type": "MiniMaxH3ReferenceToVideo",
"inputs": {
"clip": ["13", 0],
"vae": ["11", 0],
"audio_vae": ["24", 0],
"prompt": "Use <Picture 1> as the sole character and style reference. A young woman walks slowly through a quiet garden at golden hour, looking toward the camera, then turning to watch leaves drift past. Soft cinematic lighting, shallow depth of field, gentle breeze in her hair.\n\nAudio: quiet outdoor ambience, faint birds, a soft piano motif, no lyrics.",
"width": 864,
"height": 480,
"length": 73,
"ref_image_size": "match",
"ref_images.ref_image_0": ["137", 0]
}
},
"17": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "res_multistep"
}
},
"9": {
"class_type": "BasicScheduler",
"inputs": {
"model": ["6", 0],
"scheduler": "beta",
"steps": 8,
"denoise": 1.0
}
},
"15": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 1101
}
},
"16": {
"class_type": "BasicGuider",
"inputs": {
"model": ["6", 0],
"conditioning": ["136", 0]
}
},
"14": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["15", 0],
"guider": ["16", 0],
"sampler": ["17", 0],
"sigmas": ["9", 0],
"latent_image": ["136", 1]
}
},
"10": {
"class_type": "VAEDecode",
"inputs": {
"samples": ["14", 0],
"vae": ["11", 0]
}
},
"23": {
"class_type": "VAEDecodeAudio",
"inputs": {
"samples": ["14", 0],
"vae": ["24", 0]
}
},
"91": {
"class_type": "CreateVideo",
"inputs": {
"images": ["10", 0],
"audio": ["23", 0],
"fps": 24,
"bit_depth": 8
}
},
"92": {
"class_type": "SaveVideo",
"inputs": {
"video": ["91", 0],
"filename_prefix": "video/MiniMax_H3_sgld_r2v",
"format": "auto",
"codec": "auto"
}
}
}
@@ -0,0 +1,133 @@
{
"6": {
"class_type": "SGLDUNETLoader",
"inputs": {
"unet_name": "minimax_h3_fl2va_bf16.safetensors",
"weight_dtype": "default",
"sgld_options": ["52", 0]
},
"_meta": {
"title": "SGLDiffusion UNET Loader"
}
},
"52": {
"class_type": "SGLDOptions",
"inputs": {
"model_type": "minimax_h3",
"enable_torch_compile": false,
"num_gpus": 1,
"tp_size": -1,
"sp_degree": -1,
"ulysses_degree": -1,
"ring_degree": -1,
"dp_size": 1,
"dp_degree": 1,
"enable_cfg_parallel": false,
"attention_backend": ""
},
"_meta": {
"title": "SGLDiffusion Options"
}
},
"13": {
"class_type": "CLIPLoader",
"inputs": {
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
"type": "minimax",
"device": "default"
}
},
"11": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
}
},
"24": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_audio_vae_fp32.safetensors"
}
},
"104": {
"class_type": "MiniMaxH3ImageToVideo",
"inputs": {
"clip": ["13", 0],
"vae": ["11", 0],
"prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.",
"width": 864,
"height": 480,
"length": 73
}
},
"17": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "res_multistep"
}
},
"9": {
"class_type": "BasicScheduler",
"inputs": {
"model": ["6", 0],
"scheduler": "simple",
"steps": 8,
"denoise": 1.0
}
},
"15": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 1101
}
},
"16": {
"class_type": "BasicGuider",
"inputs": {
"model": ["6", 0],
"conditioning": ["104", 0]
}
},
"14": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["15", 0],
"guider": ["16", 0],
"sampler": ["17", 0],
"sigmas": ["9", 0],
"latent_image": ["104", 1]
}
},
"10": {
"class_type": "VAEDecode",
"inputs": {
"samples": ["14", 0],
"vae": ["11", 0]
}
},
"23": {
"class_type": "VAEDecodeAudio",
"inputs": {
"samples": ["14", 0],
"vae": ["24", 0]
}
},
"91": {
"class_type": "CreateVideo",
"inputs": {
"images": ["10", 0],
"audio": ["23", 0],
"fps": 24,
"bit_depth": 8
}
},
"92": {
"class_type": "SaveVideo",
"inputs": {
"video": ["91", 0],
"filename_prefix": "video/MiniMax_H3_sgld_t2v",
"format": "auto",
"codec": "auto"
}
}
}
@@ -0,0 +1,241 @@
{
"6": {
"class_type": "SGLDUNETLoader",
"inputs": {
"unet_name": "minimax_h3_fl2va_bf16.safetensors",
"weight_dtype": "default",
"sgld_options": ["52", 0]
}
},
"7": {
"class_type": "SGLDLoraLoader",
"inputs": {
"model": ["6", 0],
"lora_name": "minimax_h3_fl2v_lightx2v_turbo_4step_v0.1_comfy.safetensors",
"strength_model": 1.0,
"nickname": "lightx2v",
"target": "all"
}
},
"52": {
"class_type": "SGLDOptions",
"inputs": {
"model_type": "minimax_h3",
"enable_torch_compile": false,
"num_gpus": 1,
"tp_size": -1,
"sp_degree": -1,
"ulysses_degree": -1,
"ring_degree": -1,
"dp_size": 1,
"dp_degree": 1,
"enable_cfg_parallel": false,
"attention_backend": "",
"dit_layerwise_offload": false,
"enable_cache_dit": false,
"quantization": "",
"transformer_weights_path": ""
}
},
"13": {
"class_type": "CLIPLoader",
"inputs": {
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
"type": "minimax",
"device": "default"
}
},
"11": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
}
},
"24": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_audio_vae_fp32.safetensors"
}
},
"104": {
"class_type": "MiniMaxH3ImageToVideo",
"inputs": {
"clip": ["13", 0],
"vae": ["11", 0],
"prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.",
"width": 768,
"height": 448,
"length": 39
}
},
"17": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "res_multistep"
}
},
"9": {
"class_type": "BasicScheduler",
"inputs": {
"model": ["7", 0],
"scheduler": "simple",
"steps": 4,
"denoise": 1.0
}
},
"15": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 7703
}
},
"16": {
"class_type": "BasicGuider",
"inputs": {
"model": ["7", 0],
"conditioning": ["104", 0]
}
},
"14": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["15", 0],
"guider": ["16", 0],
"sampler": ["17", 0],
"sigmas": ["9", 0],
"latent_image": ["104", 1]
}
},
"93": {
"class_type": "VAEDecode",
"inputs": {
"samples": ["14", 0],
"vae": ["11", 0]
}
},
"23": {
"class_type": "VAEDecodeAudio",
"inputs": {
"samples": ["14", 0],
"vae": ["24", 0]
}
},
"94": {
"class_type": "CreateVideo",
"inputs": {
"images": ["93", 0],
"audio": ["23", 0],
"fps": 24,
"bit_depth": 8
}
},
"95": {
"class_type": "SaveVideo",
"inputs": {
"video": ["94", 0],
"filename_prefix": "video/MiniMax_H3_sgld_t2v_upscaler_low",
"format": "auto",
"codec": "auto"
}
},
"201": {
"class_type": "LTXVSeparateAVLatent",
"inputs": {
"av_latent": ["14", 0]
}
},
"202": {
"class_type": "MinimaxH3LatentUpscaler3D",
"inputs": {
"latent": ["201", 0],
"model_name": "minimax_h3_latent_upscaler_3d_fp16.safetensors",
"mode": "scale by multiplier",
"mode.scale": 2.0,
"align": 32,
"device": "cuda",
"precision": "fp16"
}
},
"203": {
"class_type": "LTXVConcatAVLatent",
"inputs": {
"video_latent": ["202", 0],
"audio_latent": ["201", 1]
}
},
"204": {
"class_type": "MiniMaxH3ImageToVideo",
"inputs": {
"clip": ["13", 0],
"vae": ["11", 0],
"prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.",
"width": 1536,
"height": 896,
"length": 39
}
},
"209": {
"class_type": "BasicScheduler",
"inputs": {
"model": ["7", 0],
"scheduler": "simple",
"steps": 4,
"denoise": 0.45
}
},
"215": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 7704
}
},
"216": {
"class_type": "BasicGuider",
"inputs": {
"model": ["7", 0],
"conditioning": ["204", 0]
}
},
"214": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["215", 0],
"guider": ["216", 0],
"sampler": ["17", 0],
"sigmas": ["209", 0],
"latent_image": ["203", 0]
}
},
"10": {
"class_type": "VAEDecode",
"inputs": {
"samples": ["214", 0],
"vae": ["11", 0]
}
},
"96": {
"class_type": "VAEDecodeAudio",
"inputs": {
"samples": ["214", 0],
"vae": ["24", 0]
}
},
"91": {
"class_type": "CreateVideo",
"inputs": {
"images": ["10", 0],
"audio": ["96", 0],
"fps": 24,
"bit_depth": 8
}
},
"92": {
"class_type": "SaveVideo",
"inputs": {
"video": ["91", 0],
"filename_prefix": "video/MiniMax_H3_sgld_t2v_upscaler",
"format": "auto",
"codec": "auto"
}
}
}
@@ -104,6 +104,7 @@ class MiniMaxH3DiTArchConfig(DiTArchConfig):
norm_eps: float = 1e-5
qk_norm_eps: float = 1e-5
final_norm_eps: float = 1e-5
qkv_checkpoint_grouped: bool = True
checkpoint_uses_diffusers_layout: bool = False
adaln_affine_input_dim: int | None = None
has_gate_compress: bool = False
@@ -33,6 +33,11 @@ class MiniMaxH3AudioVAEConfig(VAEConfig):
super().update_model_arch(model_dict)
def post_init(self) -> None:
if (
self.arch_config.latents_mean is None
and self.arch_config.latents_std is None
):
return
validate_minimax_h3_vae_latent_stats(
self.arch_config,
component_name="audio_vae",
@@ -69,6 +69,12 @@ class MiniMaxH3VideoVAEConfig(VAEConfig):
def post_init(self) -> None:
self.resolved_parallel_decode_mode()
if (
self.arch_config.latents_mean is None
and self.arch_config.latents_std is None
):
# ComfyUI DiT-forward never loads this VAE; stats stay unset.
return
validate_minimax_h3_vae_latent_stats(
self.arch_config,
component_name="video_vae",
@@ -54,6 +54,10 @@ from sglang.multimodal_gen.runtime.disaggregation.transport.protocol import (
encode_transfer_msg,
is_transfer_message,
)
from sglang.multimodal_gen.runtime.distributed.ipc_cuda import (
attach_cuda_tensors,
detach_cuda_tensors,
)
from sglang.multimodal_gen.runtime.distributed.utils import broadcast_pyobj
from sglang.multimodal_gen.runtime.entrypoints.utils import expand_request_outputs
from sglang.multimodal_gen.runtime.pipelines_core import Req
@@ -793,6 +797,31 @@ class SchedulerDisaggMixin:
return data
def _broadcast_recv_reqs(self: Scheduler, recv_reqs):
"""ComfyUI multi-rank recv: pickle the Req skeleton, NCCL the CUDA tensors.
The general SP/CFG/TP path stays in ``Scheduler.recv_reqs`` as the
original whole-list ``broadcast_pyobj``. This helper is only the
ComfyUI overlay and does not use disagg extract.
"""
is_rank0 = self.gpu_id == 0
if is_rank0:
assert recv_reqs is not None, "rank 0 must pass the ZMQ poll result"
skeleton, tensors = detach_cuda_tensors(recv_reqs)
else:
skeleton, tensors = None, None
skeleton = self._broadcast_to_all_ranks(skeleton)
tensors = self._broadcast_tensor_dict_to_all_ranks(tensors)
if is_rank0:
return recv_reqs
if not skeleton:
return []
local_device = torch.device(
f"{current_platform.device_type}:{self.worker.local_rank}"
)
return attach_cuda_tensors(skeleton, tensors or {}, device=local_device)
def _is_multi_rank(self: Scheduler) -> bool:
sa = self.server_args
return sa.sp_degree != 1 or sa.tp_size > 1 or sa.enable_cfg_parallel
@@ -0,0 +1,288 @@
# SPDX-License-Identifier: Apache-2.0
"""Keep CUDA tensors off the pickle path for local scheduler hops.
Two hops share the same tree walk:
- ComfyUI / DiffGenerator ↔ rank-0: replace tensors with ``CudaIpcRef``
handles before pickle, rebuild with ``UntypedStorage._new_shared_cuda``
- ComfyUI multi-rank recv: detach tensors for NCCL, pickle the skeleton,
attach them again (not the disagg extract path)
"""
from __future__ import annotations
import copy
import dataclasses
from collections import OrderedDict
from dataclasses import dataclass
from typing import Any
import torch
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
logger = init_logger(__name__)
# Producer-side retain: CUDA IPC cannot open a handle in the same process,
# and the allocation must stay alive until the consumer maps it.
# Do not LRU-evict: a live handle that is dropped before the peer maps it
# silently corrupts the hop.
_PRODUCER_TENSORS: OrderedDict[tuple[bytes, int], torch.Tensor] = OrderedDict()
_WARNED_ASYNC_ALLOC = False
def _producer_key(handle: bytes, storage_offset_bytes: int) -> tuple[bytes, int]:
return (handle, storage_offset_bytes)
def _retain_producer_tensor(
handle: bytes, storage_offset_bytes: int, tensor: torch.Tensor
) -> None:
key = _producer_key(handle, storage_offset_bytes)
_PRODUCER_TENSORS[key] = tensor
_PRODUCER_TENSORS.move_to_end(key)
def release_retained_producer_tensors() -> None:
"""Drop producer-side IPC retains after the peer has mapped them."""
_PRODUCER_TENSORS.clear()
@dataclass
class CudaIpcRef:
"""Picklable CUDA IPC handle plus enough metadata to rebuild the tensor."""
dtype: str
shape: tuple[int, ...]
stride: tuple[int, ...]
storage_offset: int
device_index: int
handle: bytes
storage_size_bytes: int
storage_offset_bytes: int
ref_counter_handle: bytes
ref_counter_offset: int
event_handle: bytes
event_sync_required: bool
@classmethod
def from_tensor(cls, tensor: torch.Tensor) -> CudaIpcRef:
if not tensor.is_cuda:
raise TypeError("CudaIpcRef only shares CUDA tensors")
tensor = tensor.detach().contiguous()
(
device_index,
handle,
storage_size_bytes,
storage_offset_bytes,
ref_counter_handle,
ref_counter_offset,
event_handle,
event_sync_required,
) = tensor.untyped_storage()._share_cuda_()
ref = cls(
dtype=str(tensor.dtype).removeprefix("torch."),
shape=tuple(tensor.shape),
stride=tuple(tensor.stride()),
storage_offset=int(tensor.storage_offset()),
device_index=int(device_index),
handle=handle,
storage_size_bytes=int(storage_size_bytes),
storage_offset_bytes=int(storage_offset_bytes),
ref_counter_handle=ref_counter_handle,
ref_counter_offset=int(ref_counter_offset),
event_handle=event_handle,
event_sync_required=bool(event_sync_required),
)
_retain_producer_tensor(handle, int(storage_offset_bytes), tensor)
return ref
def materialize(self) -> torch.Tensor:
dtype = getattr(torch, self.dtype)
if self.handle is None or self.storage_size_bytes == 0:
return torch.empty(
self.shape, dtype=dtype, device=f"cuda:{self.device_index}"
)
local = _PRODUCER_TENSORS.pop(
_producer_key(self.handle, self.storage_offset_bytes), None
)
if local is not None:
return local.detach().clone()
torch.cuda._lazy_init()
storage = torch.UntypedStorage._new_shared_cuda(
self.device_index,
self.handle,
self.storage_size_bytes,
self.storage_offset_bytes,
self.ref_counter_handle,
self.ref_counter_offset,
self.event_handle,
self.event_sync_required,
)
typed = torch.storage.TypedStorage(
wrap_storage=storage, dtype=dtype, _internal=True
)
mapped = torch._utils._rebuild_tensor(
typed, self.storage_offset, self.shape, self.stride
)
# Own a private copy so the IPC mapping can be released immediately.
return mapped.clone()
def spill_cuda_tensors(value: Any, *, in_place: bool = False) -> Any:
"""Replace CUDA tensors with ``CudaIpcRef`` handles.
By default dataclasses are shallow-copied so the caller's tensors stay
intact (needed on the client, which still holds the original ``Req``).
Scheduler replies can pass ``in_place=True``.
"""
return _map_tree(value, _spill_one, copy_dataclasses=not in_place)
def materialize_cuda_refs(value: Any) -> Any:
"""Rebuild CUDA tensors from ``CudaIpcRef`` handles, in place on dataclasses."""
return _map_tree(value, _materialize_one, copy_dataclasses=False)
def _spill_one(value: Any) -> Any:
if isinstance(value, torch.Tensor) and value.is_cuda:
try:
return CudaIpcRef.from_tensor(value)
except RuntimeError as exc:
# cudaMallocAsync (ComfyUI default) cannot export IPC handles.
# Leave the tensor in place so pickle falls back to a host copy.
msg = str(exc)
if "shareIpcHandle" not in msg and "cudaMallocAsync" not in msg:
raise
global _WARNED_ASYNC_ALLOC
if not _WARNED_ASYNC_ALLOC:
_WARNED_ASYNC_ALLOC = True
logger.warning(
"CUDA IPC export failed (%s); falling back to a host pickle "
"copy. ComfyUI's default cudaMallocAsync pool cannot export "
"IPC handles.",
exc,
)
return value
def _materialize_one(value: Any) -> Any:
if isinstance(value, CudaIpcRef):
return value.materialize()
return value
def _map_tree(value: Any, fn, copy_dataclasses: bool) -> Any:
replaced = fn(value)
if replaced is not value:
return replaced
if isinstance(value, list):
return [_map_tree(item, fn, copy_dataclasses) for item in value]
if isinstance(value, tuple):
return tuple(_map_tree(item, fn, copy_dataclasses) for item in value)
if isinstance(value, dict):
return {
key: _map_tree(item, fn, copy_dataclasses) for key, item in value.items()
}
if dataclasses.is_dataclass(value) and not isinstance(value, type):
if copy_dataclasses:
value = copy.copy(value)
for field in dataclasses.fields(value):
current = getattr(value, field.name, None)
updated = _map_tree(current, fn, copy_dataclasses)
if updated is not current:
setattr(value, field.name, updated)
return value
return value
_SEP = "\x1f"
def detach_cuda_tensors(value: Any) -> tuple[Any, dict[str, torch.Tensor]]:
"""Copy the tree, replace CUDA tensors with ``None``, collect them by path."""
tensors: dict[str, torch.Tensor] = {}
skeleton = _detach(value, "", tensors)
return skeleton, tensors
def attach_cuda_tensors(
value: Any,
tensors: dict[str, torch.Tensor],
device: torch.device | None = None,
) -> Any:
"""Put detached CUDA tensors back onto the skeleton."""
if device is not None:
moved: dict[str, torch.Tensor] = {}
for key, tensor in tensors.items():
moved[key] = (
tensor
if tensor.device == device
else tensor.to(device, non_blocking=True)
)
tensors = moved
return _attach(value, "", tensors)
def _join(prefix: str, part: str) -> str:
return part if not prefix else f"{prefix}{_SEP}{part}"
def _detach(value: Any, prefix: str, tensors: dict[str, torch.Tensor]) -> Any:
if isinstance(value, torch.Tensor) and value.is_cuda:
tensors[prefix] = value
return None
if isinstance(value, list):
return [
_detach(item, _join(prefix, str(i)), tensors)
for i, item in enumerate(value)
]
if isinstance(value, tuple):
return tuple(
_detach(item, _join(prefix, str(i)), tensors)
for i, item in enumerate(value)
)
if isinstance(value, dict):
return {
key: _detach(item, _join(prefix, str(key)), tensors)
for key, item in value.items()
}
if dataclasses.is_dataclass(value) and not isinstance(value, type):
value = copy.copy(value)
for field in dataclasses.fields(value):
current = getattr(value, field.name, None)
updated = _detach(current, _join(prefix, field.name), tensors)
if updated is not current:
setattr(value, field.name, updated)
return value
return value
def _attach(value: Any, prefix: str, tensors: dict[str, torch.Tensor]) -> Any:
if prefix in tensors:
return tensors[prefix]
if isinstance(value, list):
return [
_attach(item, _join(prefix, str(i)), tensors)
for i, item in enumerate(value)
]
if isinstance(value, tuple):
return tuple(
_attach(item, _join(prefix, str(i)), tensors)
for i, item in enumerate(value)
)
if isinstance(value, dict):
return {
key: _attach(item, _join(prefix, str(key)), tensors)
for key, item in value.items()
}
if dataclasses.is_dataclass(value) and not isinstance(value, type):
for field in dataclasses.fields(value):
current = getattr(value, field.name, None)
updated = _attach(current, _join(prefix, field.name), tensors)
if updated is not current:
setattr(value, field.name, updated)
return value
return value
@@ -0,0 +1,34 @@
# SPDX-License-Identifier: Apache-2.0
"""ComfyUI single-file DiT checkpoints.
``spec`` holds the shared checkpoint description and load path. Each sibling
module registers one DiT family. Importing this package registers every spec.
"""
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( # noqa: F401
flux,
minimax_h3,
qwen_image,
zimage,
)
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
ComfyUICheckpointSpec,
ParamNamesMapping,
WeightIterator,
get_comfyui_checkpoint_spec,
get_registered_comfyui_pipeline_names,
is_comfyui_single_file,
load_comfyui_transformer,
register_comfyui_checkpoint,
)
__all__ = [
"ComfyUICheckpointSpec",
"ParamNamesMapping",
"WeightIterator",
"get_comfyui_checkpoint_spec",
"get_registered_comfyui_pipeline_names",
"is_comfyui_single_file",
"load_comfyui_transformer",
"register_comfyui_checkpoint",
]
@@ -0,0 +1,262 @@
# SPDX-License-Identifier: Apache-2.0
"""ComfyUI Flux checkpoint spec."""
import re
from typing import Any
import torch
from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
ComfyUICheckpointSpec,
WeightIterator,
register_comfyui_checkpoint,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
logger = init_logger(__name__)
def _build_dit_config(server_args: ServerArgs) -> FluxConfig:
dit_config = getattr(server_args.pipeline_config, "dit_config", None)
if not isinstance(dit_config, FluxConfig):
dit_config = FluxConfig()
server_args.pipeline_config.dit_config = dit_config
# ComfyUI Flux checkpoints always carry guidance_in weights.
dit_config.arch_config.guidance_embeds = True
return dit_config
def _split_sizes(dit_config: FluxConfig) -> tuple[int, int]:
arch_config = dit_config.arch_config
hidden_size = arch_config.num_attention_heads * arch_config.attention_head_dim
mlp_hidden_dim = int(hidden_size * getattr(arch_config, "mlp_ratio", 4.0))
return 3 * hidden_size, mlp_hidden_dim
def _split_qkv(
tensor: torch.Tensor, hidden_size: int
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
return (
tensor[:hidden_size],
tensor[hidden_size : 2 * hidden_size],
tensor[2 * hidden_size : 3 * hidden_size],
)
def _swap_halves(tensor: torch.Tensor) -> torch.Tensor:
"""ComfyUI emits [shift, scale]; AdaLayerNormContinuous expects [scale, shift]."""
half = tensor.shape[0] // 2
return torch.cat([tensor[half:], tensor[:half]], dim=0)
def _convert_weights(weights: WeightIterator, dit_config: Any) -> WeightIterator:
qkv_size, mlp_hidden_dim = _split_sizes(dit_config)
has_guidance_embeds = dit_config.arch_config.guidance_embeds
hidden_size = qkv_size // 3
for name, tensor in weights:
if not has_guidance_embeds and name.startswith("guidance_in."):
continue
match = re.match(
r"double_blocks\.(\d+)\.(img_attn|txt_attn)\.qkv\.(weight|bias)$", name
)
if match:
block_idx, attn_type, param_type = match.groups()
if tensor.shape[0] < qkv_size:
logger.warning(
"%s shape %s smaller than expected qkv size %d, skipping",
name,
tensor.shape,
qkv_size,
)
continue
q, k, v = _split_qkv(tensor, hidden_size)
prefix = f"transformer_blocks.{block_idx}.attn"
if attn_type == "img_attn":
yield f"{prefix}.to_q.{param_type}", q
yield f"{prefix}.to_k.{param_type}", k
yield f"{prefix}.to_v.{param_type}", v
else:
yield f"{prefix}.add_q_proj.{param_type}", q
yield f"{prefix}.add_k_proj.{param_type}", k
yield f"{prefix}.add_v_proj.{param_type}", v
continue
match = re.match(r"single_blocks\.(\d+)\.linear1\.(weight|bias)$", name)
if match:
block_idx, param_type = match.groups()
expected_size = qkv_size + mlp_hidden_dim
if tensor.shape[0] < expected_size:
logger.warning(
"linear1.%s shape %s doesn't match expected size %d, skipping",
param_type,
tensor.shape,
expected_size,
)
continue
q, k, v = _split_qkv(tensor[:qkv_size], hidden_size)
prefix = f"single_transformer_blocks.{block_idx}"
yield f"{prefix}.attn.to_q.{param_type}", q
yield f"{prefix}.attn.to_k.{param_type}", k
yield f"{prefix}.attn.to_v.{param_type}", v
yield f"{prefix}.proj_mlp.{param_type}", tensor[qkv_size:]
continue
if name in (
"final_layer.adaLN_modulation.1.weight",
"final_layer.adaLN_modulation.1.bias",
):
yield name, _swap_halves(tensor)
continue
yield name, tensor
# ComfyUI names differ from SGLang's diffusers-style names. Fused tensors that
# _convert_weights already split are emitted under their SGLang names, so only
# the untouched ones need an entry here.
_PARAM_NAMES_MAPPING = {
r"double_blocks\.(\d+)\.img_attn\.proj\.(weight|bias)$": (
r"transformer_blocks.\1.attn.to_out.0.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_attn\.proj\.(weight|bias)$": (
r"transformer_blocks.\1.attn.to_add_out.\2",
None,
None,
),
r"double_blocks\.(\d+)\.img_attn\.norm\.query_norm\.scale$": (
r"transformer_blocks.\1.attn.norm_q.weight",
None,
None,
),
r"double_blocks\.(\d+)\.img_attn\.norm\.key_norm\.scale$": (
r"transformer_blocks.\1.attn.norm_k.weight",
None,
None,
),
r"double_blocks\.(\d+)\.txt_attn\.norm\.query_norm\.scale$": (
r"transformer_blocks.\1.attn.norm_added_q.weight",
None,
None,
),
r"double_blocks\.(\d+)\.txt_attn\.norm\.key_norm\.scale$": (
r"transformer_blocks.\1.attn.norm_added_k.weight",
None,
None,
),
r"double_blocks\.(\d+)\.img_mlp\.0\.(weight|bias)$": (
r"transformer_blocks.\1.ff.net.0.proj.\2",
None,
None,
),
r"double_blocks\.(\d+)\.img_mlp\.2\.(weight|bias)$": (
r"transformer_blocks.\1.ff.net.2.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_mlp\.0\.(weight|bias)$": (
r"transformer_blocks.\1.ff_context.net.0.proj.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_mlp\.2\.(weight|bias)$": (
r"transformer_blocks.\1.ff_context.net.2.\2",
None,
None,
),
r"double_blocks\.(\d+)\.img_mod\.lin\.(weight|bias)$": (
r"transformer_blocks.\1.norm1.linear.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_mod\.lin\.(weight|bias)$": (
r"transformer_blocks.\1.norm1_context.linear.\2",
None,
None,
),
r"single_blocks\.(\d+)\.linear2\.(weight|bias)$": (
r"single_transformer_blocks.\1.proj_out.\2",
None,
None,
),
r"single_blocks\.(\d+)\.norm\.query_norm\.scale$": (
r"single_transformer_blocks.\1.attn.norm_q.weight",
None,
None,
),
r"single_blocks\.(\d+)\.norm\.key_norm\.scale$": (
r"single_transformer_blocks.\1.attn.norm_k.weight",
None,
None,
),
r"single_blocks\.(\d+)\.modulation\.lin\.(weight|bias)$": (
r"single_transformer_blocks.\1.norm.linear.\2",
None,
None,
),
r"^time_in\.in_layer\.(weight|bias)$": (
r"time_text_embed.timestep_embedder.linear_1.\1",
None,
None,
),
r"^time_in\.out_layer\.(weight|bias)$": (
r"time_text_embed.timestep_embedder.linear_2.\1",
None,
None,
),
r"^txt_in\.(weight|bias)$": (r"context_embedder.\1", None, None),
r"^vector_in\.in_layer\.(weight|bias)$": (
r"time_text_embed.text_embedder.linear_1.\1",
None,
None,
),
r"^vector_in\.out_layer\.(weight|bias)$": (
r"time_text_embed.text_embedder.linear_2.\1",
None,
None,
),
r"^final_layer\.linear\.(weight|bias)$": (r"proj_out.\1", None, None),
r"^final_layer\.norm_final\.(weight|bias)$": (r"norm_out.\1", None, None),
r"^final_layer\.adaLN_modulation\.1\.(weight|bias)$": (
r"norm_out.linear.\1",
None,
None,
),
r"^img_in\.(weight|bias)$": (r"x_embedder.\1", None, None),
r"^guidance_in\.in_layer\.(weight|bias)$": (
r"time_text_embed.guidance_embedder.linear_1.\1",
None,
None,
),
r"^guidance_in\.out_layer\.(weight|bias)$": (
r"time_text_embed.guidance_embedder.linear_2.\1",
None,
None,
),
}
register_comfyui_checkpoint(
"FluxPipeline",
ComfyUICheckpointSpec(
dit_cls_name="FluxTransformer2DModel",
build_dit_config=_build_dit_config,
param_names_mapping=_PARAM_NAMES_MAPPING,
convert_weights=_convert_weights,
# ComfyUI Flux checkpoints ship without the optional attention biases.
strict=False,
# FluxConfig's own mapping reads the same BFL names this spec does, but
# targets a different parameter layout (fused to_qkv, ff.linear_in,
# time_guidance_embed) than FluxTransformer2DModel exposes. Layering the
# two rewrites correctly resolved names into ones the model lacks.
inherit_config_mapping=False,
),
)
@@ -0,0 +1,35 @@
# SPDX-License-Identifier: Apache-2.0
"""ComfyUI MiniMax-H3 checkpoint spec.
The Comfy-Org BF16 FL2VA file uses ComfyUI parameter names and already stores
fused QKV as ``[q_all, k_all, v_all]``. Identity name mapping is enough, but
the DiT's default HF weight loader must not run the grouped→concat reorder.
"""
from sglang.multimodal_gen.configs.models.dits.minimax_h3 import MiniMaxH3DiTConfig
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
ComfyUICheckpointSpec,
register_comfyui_checkpoint,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
def _build_dit_config(server_args: ServerArgs) -> MiniMaxH3DiTConfig:
dit_config = getattr(server_args.pipeline_config, "dit_config", None)
if not isinstance(dit_config, MiniMaxH3DiTConfig):
dit_config = MiniMaxH3DiTConfig()
server_args.pipeline_config.dit_config = dit_config
# ComfyUI / Comfy-Org files match the runtime QKV layout. Native
# ``sglang serve`` keeps the default grouped HF reorder.
dit_config.arch_config.qkv_checkpoint_grouped = False
return dit_config
register_comfyui_checkpoint(
"MiniMaxH3Pipeline",
ComfyUICheckpointSpec(
dit_cls_name="MiniMaxH3DiTModel",
build_dit_config=_build_dit_config,
inherit_config_mapping=True,
),
)
@@ -0,0 +1,57 @@
# SPDX-License-Identifier: Apache-2.0
"""ComfyUI Qwen-Image checkpoint specs (text-to-image and edit)."""
from collections.abc import Callable
from sglang.multimodal_gen.configs.models.dits.qwenimage import (
QwenImageArchConfig,
QwenImageDitConfig,
)
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
ComfyUICheckpointSpec,
register_comfyui_checkpoint,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
def _dit_config_builder(
zero_cond_t: bool,
) -> Callable[[ServerArgs], QwenImageDitConfig]:
def build(server_args: ServerArgs) -> QwenImageDitConfig:
# ComfyUI checkpoints carry no config, so the architecture is pinned here.
dit_config = QwenImageDitConfig(
arch_config=QwenImageArchConfig(
patch_size=2,
in_channels=64,
out_channels=16,
num_layers=60,
attention_head_dim=128,
num_attention_heads=24,
joint_attention_dim=3584,
pooled_projection_dim=768,
guidance_embeds=False,
axes_dims_rope=(16, 56, 56),
zero_cond_t=zero_cond_t,
)
)
server_args.pipeline_config.dit_config = dit_config
return dit_config
return build
_PARAM_NAMES_MAPPING = {r"^model\.diffusion_model\.(.*)$": (r"\1", None, None)}
for _pipeline_name, _zero_cond_t in (
("QwenImagePipeline", False),
("QwenImageEditPlusPipeline", True),
):
register_comfyui_checkpoint(
_pipeline_name,
ComfyUICheckpointSpec(
dit_cls_name="QwenImageTransformer2DModel",
build_dit_config=_dit_config_builder(_zero_cond_t),
param_names_mapping=_PARAM_NAMES_MAPPING,
),
)
@@ -0,0 +1,304 @@
# SPDX-License-Identifier: Apache-2.0
"""ComfyUI single-file DiT checkpoints: spec, registry, and load.
ComfyUI ships a DiT as one ``.safetensors`` (or a GGUF override) with no
``model_index.json`` and its own parameter names. A spec supplies what the
shared loader cannot infer: which DiT config to build, how names map onto
SGLang, and how to reshape tensors whose layout differs. Per-model specs live
in the sibling modules of this package.
Everything else -- meta-device init, FSDP sharding, quantization, CPU
offload -- goes through the regular transformer load path.
"""
from __future__ import annotations
import os
from collections.abc import Callable, Iterator
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any
import torch
from sglang.multimodal_gen.runtime.distributed import get_local_torch_device
from sglang.multimodal_gen.runtime.loader.fsdp_load import maybe_load_fsdp_model
from sglang.multimodal_gen.runtime.loader.weight_utils import (
safetensors_weights_iterator,
)
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
from sglang.multimodal_gen.runtime.server_args import ServerArgs
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
from sglang.multimodal_gen.runtime.utils.precision import resolve_precision
if TYPE_CHECKING:
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
ComposedPipelineBase,
)
logger = init_logger(__name__)
WeightIterator = Iterator[tuple[str, torch.Tensor]]
# ComfyUI mapping entries reuse the param_names_mapping format:
# source_regex -> (target_template, merge_index, num_params_to_merge)
ParamNamesMapping = dict[str, tuple[str, int | None, int | None]]
@dataclass(frozen=True)
class ComfyUICheckpointSpec:
"""Per-model knowledge needed to load a ComfyUI checkpoint."""
dit_cls_name: str
build_dit_config: Callable[[ServerArgs], Any]
param_names_mapping: ParamNamesMapping = field(default_factory=dict)
# Reshapes tensors that param_names_mapping cannot express. Receives the raw
# safetensors iterator plus the built dit config, yields SGLang-shaped pairs.
convert_weights: Callable[[WeightIterator, Any], WeightIterator] | None = None
# Set False for checkpoints that legitimately omit parameters the model
# declares, such as optional biases.
strict: bool = True
# Whether to layer param_names_mapping on top of the DiT config's own
# mapping. Keep it True when the two act on different names (the config
# rules then finish the job, e.g. merging split QKV back into a fused
# parameter). Set False when both claim the same source names, since name
# mapping is applied repeatedly until it reaches a fixed point and the
# config rules would rewrite names this spec already resolved.
inherit_config_mapping: bool = True
_SPEC_REGISTRY: dict[str, ComfyUICheckpointSpec] = {}
_SPECS_DISCOVERED = False
def register_comfyui_checkpoint(
pipeline_name: str, spec: ComfyUICheckpointSpec
) -> None:
_SPEC_REGISTRY[pipeline_name] = spec
def _discover_checkpoint_specs() -> None:
global _SPECS_DISCOVERED
if _SPECS_DISCOVERED:
return
_SPECS_DISCOVERED = True
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( # noqa: F401
flux,
minimax_h3,
qwen_image,
zimage,
)
def get_comfyui_checkpoint_spec(pipeline_name: str) -> ComfyUICheckpointSpec | None:
_discover_checkpoint_specs()
return _SPEC_REGISTRY.get(pipeline_name)
def get_registered_comfyui_pipeline_names() -> list[str]:
_discover_checkpoint_specs()
return sorted(_SPEC_REGISTRY)
def is_comfyui_single_file(model_path: str) -> bool:
"""ComfyUI ships DiTs as one safetensors file with no model_index.json."""
return os.path.isfile(model_path) and model_path.endswith(".safetensors")
def _load_comfyui_gguf_transformer(
*,
model_cls,
dit_config,
gguf_file: str,
server_args: ServerArgs,
mapping: dict[str, Any],
spec,
):
from sglang.multimodal_gen.runtime.layers.quantization.gguf import GGUFConfig
from sglang.multimodal_gen.runtime.loader.gguf_weights import (
gguf_weights_iterator,
read_gguf_tensor_meta,
)
from sglang.multimodal_gen.runtime.loader.minimax_h3_weights import (
validate_minimax_h3_checkpoint_variant,
)
from sglang.multimodal_gen.runtime.loader.transformer_load_utils import (
_validate_gguf_runtime_support,
)
_validate_gguf_runtime_support(server_args, "transformer")
if spec.dit_cls_name == "MiniMaxH3DiTModel":
validate_minimax_h3_checkpoint_variant(
[gguf_file], str(server_args.model_variant or "fl2va")
)
tensor_meta = read_gguf_tensor_meta(gguf_file)
curve = tensor_meta.get("adaln_t_table")
if curve is not None:
if curve.is_quantized or len(curve.logical_shape) != 2:
raise ValueError(
"MiniMax-H3 adaln_t_table must be an unquantized 2D tensor"
)
curve_grid, time_embed_dim = curve.logical_shape
if curve_grid < 2:
raise ValueError("MiniMax-H3 adaln_t_table needs at least two rows")
dit_config.arch_config.adaln_curve_grid = curve_grid
dit_config.arch_config.time_embed_dim = time_embed_dim
quant_config = GGUFConfig(gguf_file=gguf_file, tensor_meta=tensor_meta)
packed = getattr(model_cls, "packed_modules_mapping", None)
if packed:
quant_config.packed_modules_mapping = packed
logger.info("Loading %s from GGUF file %s", spec.dit_cls_name, gguf_file)
original_mapping = model_cls.param_names_mapping
model_cls.param_names_mapping = mapping
try:
return maybe_load_fsdp_model(
model_cls=model_cls,
init_params={
"config": dit_config,
"hf_config": {},
"quant_config": quant_config,
},
weight_dir_list=[],
device=get_local_torch_device(),
hsdp_replicate_dim=server_args.hsdp_replicate_dim,
hsdp_shard_dim=server_args.hsdp_shard_dim,
component_starts_on_cpu=server_args.should_start_component_on_cpu(
"transformer"
),
pin_cpu_memory=server_args.pin_cpu_memory,
fsdp_inference=server_args.should_use_fsdp_for_component("transformer"),
param_dtype=None,
reduce_dtype=torch.float32,
output_dtype=None,
strict=False,
weights_iterator=gguf_weights_iterator(gguf_file, tensor_meta),
)
finally:
model_cls.param_names_mapping = original_mapping
def load_comfyui_transformer(
pipeline: ComposedPipelineBase,
server_args: ServerArgs,
loaded_modules: dict[str, torch.nn.Module] | None = None,
) -> dict[str, Any]:
"""Load the DiT from a single ComfyUI safetensors file.
Reuses the shared FSDP-aware transformer load path; the spec only supplies
what the file itself cannot describe. A GGUF
``--transformer-weights-path`` replaces only the DiT weights.
"""
if loaded_modules is not None and "transformer" in loaded_modules:
return {
"transformer": loaded_modules["transformer"],
"scheduler": pipeline.get_module("scheduler"),
}
spec = get_comfyui_checkpoint_spec(pipeline.pipeline_name)
if spec is None:
raise ValueError(
f"{pipeline.pipeline_name} has no ComfyUI checkpoint spec, so it cannot "
f"load a single safetensors file. Pipelines with a spec: "
f"{get_registered_comfyui_pipeline_names()}"
)
model_path = pipeline.model_path
dit_config = spec.build_dit_config(server_args)
mapping = dict(spec.param_names_mapping)
if spec.inherit_config_mapping:
mapping = {
**(dit_config.arch_config.param_names_mapping or {}),
**mapping,
}
dit_config.arch_config.param_names_mapping = mapping
model_cls, _ = ModelRegistry.resolve_model_cls(spec.dit_cls_name)
param_dtype = resolve_precision(server_args, "dit", precision_attr="dit_precision")
server_args.model_paths["transformer"] = os.path.dirname(model_path) or "."
from sglang.multimodal_gen.runtime.loader.transformer_load_utils import (
resolve_transformer_gguf_to_load,
)
gguf_file = resolve_transformer_gguf_to_load(server_args, "transformer")
if gguf_file is not None:
model = _load_comfyui_gguf_transformer(
model_cls=model_cls,
dit_config=dit_config,
gguf_file=gguf_file,
server_args=server_args,
mapping=mapping,
spec=spec,
)
else:
# Only override the iterator when tensors need reshaping; leaving it None
# keeps the rank-local checkpoint fast path available.
weights_iterator = None
if spec.convert_weights is not None:
weights_iterator = spec.convert_weights(
safetensors_weights_iterator([model_path]), dit_config
)
# GGUF already sets AdaLN curve from tensor meta. Pruned BF16
# safetensors keep the same adaln_t_table; without this the DiT is
# built as the unpruned MLP and load fails on that extra parameter.
if spec.dit_cls_name == "MiniMaxH3DiTModel":
from sglang.multimodal_gen.runtime.loader.minimax_h3_weights import (
inspect_minimax_h3_safetensors,
)
adaln_curve_shape, _ = inspect_minimax_h3_safetensors([model_path])
if adaln_curve_shape is not None:
(
dit_config.arch_config.adaln_curve_grid,
dit_config.arch_config.time_embed_dim,
) = adaln_curve_shape
logger.info(
"MiniMax-H3 ComfyUI checkpoint uses AdaLN curve %s",
adaln_curve_shape,
)
logger.info(
"Loading %s from ComfyUI checkpoint %s, param_dtype: %s",
spec.dit_cls_name,
model_path,
param_dtype,
)
# Weight loading reads param_names_mapping off the model, which inherits it
# from the class, so the ComfyUI names have to be visible for the whole load.
original_mapping = model_cls.param_names_mapping
model_cls.param_names_mapping = mapping
try:
model = maybe_load_fsdp_model(
model_cls=model_cls,
init_params={"config": dit_config, "hf_config": {}},
weight_dir_list=[model_path],
device=get_local_torch_device(),
hsdp_replicate_dim=server_args.hsdp_replicate_dim,
hsdp_shard_dim=server_args.hsdp_shard_dim,
component_starts_on_cpu=server_args.should_start_component_on_cpu(
"transformer"
),
pin_cpu_memory=server_args.pin_cpu_memory,
fsdp_inference=server_args.should_use_fsdp_for_component("transformer"),
param_dtype=param_dtype,
reduce_dtype=torch.float32,
output_dtype=None,
strict=spec.strict,
weights_iterator=weights_iterator,
)
finally:
model_cls.param_names_mapping = original_mapping
for param in model.parameters():
param.requires_grad = False
logger.info(
"Loaded transformer with %.2fB parameters",
sum(p.numel() for p in model.parameters()) / 1e9,
)
return {"transformer": model, "scheduler": pipeline.get_module("scheduler")}
@@ -0,0 +1,65 @@
# SPDX-License-Identifier: Apache-2.0
"""ComfyUI Z-Image (lumina2) checkpoint spec."""
import re
from typing import Any
from sglang.multimodal_gen.configs.models.dits.zimage import ZImageDitConfig
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
ComfyUICheckpointSpec,
WeightIterator,
register_comfyui_checkpoint,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
def _build_dit_config(server_args: ServerArgs) -> ZImageDitConfig:
dit_config = getattr(server_args.pipeline_config, "dit_config", None)
if not isinstance(dit_config, ZImageDitConfig):
dit_config = ZImageDitConfig()
server_args.pipeline_config.dit_config = dit_config
return dit_config
def _convert_weights(weights: WeightIterator, dit_config: Any) -> WeightIterator:
"""Split ComfyUI's merged attention qkv into to_q / to_k / to_v."""
arch_config = dit_config.arch_config
q_size = arch_config.dim
k_size = (
arch_config.dim // arch_config.num_attention_heads
) * arch_config.n_kv_heads
for name, tensor in weights:
match = re.match(
r"(layers|noise_refiner|context_refiner)\.(\d+)\.attention\.qkv\.(weight|bias)$",
name,
)
if not match:
yield name, tensor
continue
module_name, layer_idx, param_type = match.groups()
prefix = f"{module_name}.{layer_idx}.attention"
yield f"{prefix}.to_q.{param_type}", tensor[:q_size]
yield f"{prefix}.to_k.{param_type}", tensor[q_size : q_size + k_size]
yield f"{prefix}.to_v.{param_type}", tensor[q_size + k_size :]
_PARAM_NAMES_MAPPING = {
r"(.*)\.attention\.k_norm\.weight$": (r"\1.attention.norm_k.weight", None, None),
r"(.*)\.attention\.q_norm\.weight$": (r"\1.attention.norm_q.weight", None, None),
r"(.*)\.attention\.out\.weight$": (r"\1.attention.to_out.0.weight", None, None),
r"^final_layer\.(.*)$": (r"all_final_layer.2-1.\1", None, None),
r"^x_embedder\.(.*)$": (r"all_x_embedder.2-1.\1", None, None),
}
register_comfyui_checkpoint(
"ZImagePipeline",
ComfyUICheckpointSpec(
dit_cls_name="ZImageTransformer2DModel",
build_dit_config=_build_dit_config,
param_names_mapping=_PARAM_NAMES_MAPPING,
convert_weights=_convert_weights,
),
)
@@ -15,6 +15,7 @@ from torch.distributed.fsdp import FSDPModule
from sglang.multimodal_gen.runtime.distributed import get_tp_rank, get_tp_world_size
from sglang.multimodal_gen.runtime.layers.linear import (
ColumnParallelLinear,
MergedColumnParallelLinear,
ReplicatedLinear,
RowParallelLinear,
)
@@ -236,6 +237,17 @@ def read_rank_local_tensor(
return local_tensor
def _merged_output_sizes(actual_param: torch.nn.Parameter) -> list[int] | None:
"""Logical output widths of a fused-on-disk MergedColumnParallelLinear."""
weight_loader = actual_param.__dict__.get("weight_loader")
if not isinstance(weight_loader, MethodType):
return None
owner = weight_loader.__self__
if isinstance(owner, MergedColumnParallelLinear):
return list(owner.output_sizes)
return None
def _resolve_tp_shard_dim(
actual_param: torch.nn.Parameter,
) -> tuple[bool, int | None]:
@@ -318,6 +330,7 @@ def read_tp_local_tensor(
shard_dim: int | None,
tp_rank: int,
tp_size: int,
output_sizes: list[int] | None = None,
) -> torch.Tensor:
if shard_dim is None:
assembled_shape = assembled_source_shape(sources)
@@ -337,6 +350,37 @@ def read_tp_local_tensor(
source.merge_index if source.merge_index is not None else 0,
),
)
# Fused-on-disk [Q|K|V] / [gate|up]: each logical matrix is sharded on
# its own, then concatenated. A contiguous half of the fused tensor is
# the same shape as the TP-local parameter but the wrong values.
if (
output_sizes is not None
and len(ordered_sources) == 1
and all(size % tp_size == 0 for size in output_sizes)
):
source = ordered_sources[0]
if source.shape[shard_dim] != sum(output_sizes):
raise RuntimeError(
"Merged TP shard sizes do not cover the fused checkpoint: "
f"output_sizes={output_sizes}, source_dim={source.shape[shard_dim]}"
)
local_parts = []
offset = 0
for output_size in output_sizes:
shard_size = output_size // tp_size
slices = [slice(None)] * len(source.shape)
slices[shard_dim] = slice(
offset + tp_rank * shard_size,
offset + tp_rank * shard_size + shard_size,
)
local_parts.append(
handles[source.file_path]
.get_slice(source.param_name)[tuple(slices)]
.contiguous()
)
offset += output_size
return torch.cat(local_parts, dim=shard_dim)
local_parts = []
for source in ordered_sources:
shard_size = source.shape[shard_dim] // tp_size
@@ -380,6 +424,7 @@ def try_load_rank_local_tp_state_dict(
sources_by_target, reverse_param_names_mapping = checkpoint_sources
shard_dims: dict[str, int | None] = {}
output_sizes_by_name: dict[str, list[int] | None] = {}
for target_param_name, sources in sources_by_target.items():
meta_param = meta_sd.get(target_param_name)
if meta_param is None or isinstance(meta_param, dist_tensor.DTensor):
@@ -394,15 +439,22 @@ def try_load_rank_local_tp_state_dict(
param_dict,
target_param_name,
)
output_sizes = None
if actual_param is None:
supported, shard_dim = True, None
else:
supported, shard_dim = _resolve_tp_shard_dim(actual_param)
output_sizes = _merged_output_sizes(actual_param)
if output_sizes is not None and any(
size % tp_size for size in output_sizes
):
return None
if not supported:
return None
if tp_local_shape(sources, shard_dim, tp_size) != tuple(meta_param.shape):
return None
shard_dims[target_param_name] = shard_dim
output_sizes_by_name[target_param_name] = output_sizes
local_param_sd: dict[str, LocalTPShard] = {}
local_bytes = 0
@@ -421,6 +473,7 @@ def try_load_rank_local_tp_state_dict(
shard_dims[target_param_name],
tp_rank,
tp_size,
output_sizes=output_sizes_by_name[target_param_name],
)
local_param_sd[target_param_name] = LocalTPShard(tensor)
local_bytes += tensor.numel() * tensor.element_size()
@@ -16,6 +16,11 @@ from sglang.multimodal_gen.runtime.disaggregation.roles import RoleType
from sglang.multimodal_gen.runtime.disaggregation.scheduler_mixin import (
SchedulerDisaggMixin,
)
from sglang.multimodal_gen.runtime.distributed.ipc_cuda import (
materialize_cuda_refs,
release_retained_producer_tensors,
spill_cuda_tensors,
)
from sglang.multimodal_gen.runtime.distributed.utils import broadcast_pyobj
from sglang.multimodal_gen.runtime.entrypoints.control_requests import (
GetDisaggStatsReq,
@@ -771,6 +776,13 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag
output_batch.output
)
with self._record_return_stage(
output_batch, "Scheduler.return_result.spill_cuda"
):
# The previous reply has already been mapped: the client
# only sends the next hop after materializing the last one.
release_retained_producer_tensors()
spill_cuda_tensors(output_batch, in_place=True)
with self._record_return_stage(
output_batch, "Scheduler.return_result.pickle"
):
@@ -1183,7 +1195,9 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag
def recv_reqs(self) -> List[tuple[bytes, Any]]:
"""
For non-main schedulers, reqs are broadcasted from main using broadcast_pyobj
For non-main schedulers, reqs are broadcasted from main using
broadcast_pyobj. ``--comfyui-mode`` multi-rank instead keeps CUDA
tensors on NCCL so per-step latents do not pickle onto the gloo group.
"""
if self.receiver is not None:
try:
@@ -1208,30 +1222,37 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag
else:
recv_reqs = None
# TODO: fix this condition
if self.server_args.sp_degree != 1:
recv_reqs = broadcast_pyobj(
recv_reqs,
self.worker.sp_group.rank,
self.worker.sp_cpu_group,
src=self.worker.sp_group.ranks[0],
)
# Rebuild CUDA IPC handles on rank 0 (no-op when the payload has none).
if recv_reqs is not None:
recv_reqs = materialize_cuda_refs(recv_reqs)
if self.server_args.enable_cfg_parallel:
recv_reqs = broadcast_pyobj(
recv_reqs,
self.worker.cfg_group.rank,
self.worker.cfg_cpu_group,
src=self.worker.cfg_group.ranks[0],
)
if self.server_args.comfyui_mode and self._is_multi_rank():
recv_reqs = self._broadcast_recv_reqs(recv_reqs)
else:
# TODO: fix this condition
if self.server_args.sp_degree != 1:
recv_reqs = broadcast_pyobj(
recv_reqs,
self.worker.sp_group.rank,
self.worker.sp_cpu_group,
src=self.worker.sp_group.ranks[0],
)
if self.server_args.tp_size > 1:
recv_reqs = broadcast_pyobj(
recv_reqs,
self.worker.tp_group.rank,
self.worker.tp_cpu_group,
src=self.worker.tp_group.ranks[0],
)
if self.server_args.enable_cfg_parallel:
recv_reqs = broadcast_pyobj(
recv_reqs,
self.worker.cfg_group.rank,
self.worker.cfg_cpu_group,
src=self.worker.cfg_group.ranks[0],
)
if self.server_args.tp_size > 1:
recv_reqs = broadcast_pyobj(
recv_reqs,
self.worker.tp_group.rank,
self.worker.tp_cpu_group,
src=self.worker.tp_group.ranks[0],
)
assert recv_reqs is not None
@@ -822,8 +822,9 @@ class MiniMaxH3Attention(nn.Module):
quant_config=quant_config,
prefix=f"{prefix}.qkv_proj",
)
# Official safetensors interleave Q/K/V by head. Comfy and GGUF
# checkpoints already store [q_all, k_all, v_all].
# Official safetensors interleave Q/K/V by head. Comfy BF16, Comfy
# convrot int8, and GGUF already store [q_all, k_all, v_all].
skip_grouped_reorder = not getattr(arch, "qkv_checkpoint_grouped", True)
checkpoint_qkv_is_native = quant_config is not None and (
quant_config.get_name() == "gguf"
or quant_config.checkpoint_uses_native_qkv_layout
@@ -831,7 +832,7 @@ class MiniMaxH3Attention(nn.Module):
checkpoint_qkv_is_native = (
checkpoint_qkv_is_native or arch.checkpoint_uses_diffusers_layout
)
if not checkpoint_qkv_is_native:
if not skip_grouped_reorder and not checkpoint_qkv_is_native:
self._install_qkv_weight_loader(arch)
self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
@@ -112,7 +112,9 @@ class ComfyUIPassThroughScheduler(BaseScheduler, ConfigMixin, SchedulerMixin):
Returns:
The input sample unchanged (prev_sample = sample)
"""
# Increment step index for tracking
# DenoisingStage resets _step_index to None before each request.
if self._step_index is None:
self._step_index = 0
self._step_index += 1
# Simply return the input sample unchanged
@@ -1,701 +0,0 @@
# Copied and adapted from: https://github.com/hao-ai-lab/FastVideo
# SPDX-License-Identifier: Apache-2.0
import os
import re
from typing import Any, Generator
import torch
from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig
from sglang.multimodal_gen.runtime.distributed import get_local_torch_device
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import (
ComfyUIPassThroughScheduler,
)
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
ComposedPipelineBase,
)
from sglang.multimodal_gen.runtime.pipelines_core.stages import (
ComfyUILatentPreparationStage,
DenoisingStage,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
from sglang.multimodal_gen.runtime.utils.precision import resolve_precision
logger = init_logger(__name__)
class ComfyUIFluxPipeline(LoRAPipeline, ComposedPipelineBase):
"""
Simplified pipeline for ComfyUI integration with only denoising stage.
This pipeline requires pre-processed inputs:
- prompt_embeds: Pre-encoded text embeddings (list of tensors)
- negative_prompt_embeds: Pre-encoded negative prompt embeddings (if using CFG)
- latents: Optional initial noise latents (will be generated if not provided)
Usage:
generator = DiffGenerator.from_pretrained(
model_path="path/to/model",
pipeline_class_name="ComfyUIFluxPipeline",
device="cuda",
)
"""
pipeline_name = "ComfyUIFluxPipeline"
# Configuration classes for safetensors files without model_index.json
from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig
from sglang.multimodal_gen.configs.sample.flux import FluxSamplingParams
pipeline_config_cls = FluxPipelineConfig
sampling_params_cls = FluxSamplingParams
_required_config_modules = [
"transformer",
"scheduler",
]
def initialize_pipeline(self, server_args: ServerArgs):
"""
Initialize the pipeline with ComfyUI pass-through scheduler.
This scheduler does not modify latents, allowing ComfyUI to handle denoising.
"""
self.modules["scheduler"] = ComfyUIPassThroughScheduler(
num_train_timesteps=1000
)
if hasattr(server_args.pipeline_config, "vae_config"):
vae_config = server_args.pipeline_config.vae_config
if hasattr(vae_config, "post_init") and not hasattr(
vae_config, "_post_init_called"
):
vae_config.post_init()
logger.info(
"Called vae_config.post_init() to set spatial_compression_ratio. "
f"spatial_compression_ratio={vae_config.arch_config.spatial_compression_ratio}"
)
def load_modules(
self,
server_args: ServerArgs,
loaded_modules: dict[str, torch.nn.Module] | None = None,
) -> dict[str, Any]:
"""
Load modules for ComfyUIFluxPipeline.
If model_path is a safetensors file, load transformer directly from it
without requiring model_index.json. Otherwise, fall back to default loading.
"""
if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"):
logger.info(
"Detected safetensors file, loading transformer directly from: %s",
self.model_path,
)
return self._load_transformer_from_safetensors(server_args, loaded_modules)
else:
logger.info(
"Model path is a directory, using default loading method: %s",
self.model_path,
)
return super().load_modules(server_args, loaded_modules)
def _load_and_convert_weights_from_safetensors(
self,
model_cls: type,
dit_config: FluxConfig,
hf_config: dict,
safetensors_list: list[str],
updated_mapping: dict,
qkv_size: int,
mlp_hidden_dim: int,
has_guidance_embeds: bool,
default_dtype: torch.dtype,
) -> tuple[torch.nn.Module, dict]:
"""
Load and convert weights from safetensors file, then load them into the model.
"""
from sglang.multimodal_gen.runtime.loader.utils import (
get_param_names_mapping,
set_default_torch_dtype,
)
from sglang.multimodal_gen.runtime.loader.weight_utils import (
safetensors_weights_iterator,
)
logger.info(
"Converting ComfyUI Flux weights to SGLang format and loading model..."
)
# Create model on target device
device = get_local_torch_device()
with set_default_torch_dtype(default_dtype):
model = model_cls(**{"config": dit_config, "hf_config": hf_config})
model = model.to(device)
# Verify model has guidance_embedder if config says it should
has_guidance_embedder = hasattr(model.time_text_embed, "guidance_embedder")
if has_guidance_embeds and not has_guidance_embedder:
logger.warning(
"Config has guidance_embeds=True but model doesn't have guidance_embedder. "
"This may indicate a configuration mismatch."
)
elif not has_guidance_embeds and has_guidance_embedder:
logger.warning(
"Config has guidance_embeds=False but model has guidance_embedder. "
"This may indicate a configuration mismatch."
)
# Note: guidance_in mappings are already included in comfyui_flux_mappings above.
# If model doesn't support guidance embeddings, the weights will be filtered out
# in _convert_comfyui_weights() based on has_guidance_embeds flag.
param_names_mapping_fn = get_param_names_mapping(updated_mapping)
weight_iterator = safetensors_weights_iterator(safetensors_list)
converted_weights = self._convert_comfyui_weights(
weight_iterator=weight_iterator,
qkv_size=qkv_size,
mlp_hidden_dim=mlp_hidden_dim,
has_guidance_embeds=has_guidance_embeds,
)
model_state_dict = model.state_dict()
missing_keys = set(model_state_dict.keys())
unexpected_keys = []
loaded_count = 0
reverse_param_names_mapping = {}
# Handle merged parameters (collect all parts before merging)
from collections import defaultdict
to_merge_params = defaultdict(dict)
# Process weights incrementally: load immediately after conversion
for source_name, tensor in converted_weights:
target_name, merge_index, num_params_to_merge = param_names_mapping_fn(
source_name
)
reverse_param_names_mapping[target_name] = (
source_name,
merge_index,
num_params_to_merge,
)
if merge_index is not None:
# Collect parts for merging
to_merge_params[target_name][merge_index] = tensor
if len(to_merge_params[target_name]) == num_params_to_merge:
# All parts collected, merge them
sorted_tensors = [
to_merge_params[target_name][i]
for i in range(num_params_to_merge)
]
merged_tensor = torch.cat(sorted_tensors, dim=0)
# Load immediately after merging
if target_name in model_state_dict:
param = model_state_dict[target_name]
loaded_tensor = merged_tensor.to(
device=param.device, dtype=param.dtype
)
param.data.copy_(loaded_tensor)
missing_keys.discard(target_name)
loaded_count += 1
del merged_tensor, loaded_tensor
else:
unexpected_keys.append(target_name)
# Clear merged parts
del to_merge_params[target_name]
for t in sorted_tensors:
del t
else:
# Direct mapping, load immediately
if target_name in model_state_dict:
param = model_state_dict[target_name]
# Check shape compatibility
if tensor.shape != param.shape:
logger.warning(
f"Shape mismatch for {target_name}: "
f"loaded {tensor.shape} vs model {param.shape}, skipping. "
f"Source: {source_name}"
)
unexpected_keys.append(target_name)
del tensor
continue
# Debug logging for norm_out.linear to verify mapping
if (
"norm_out.linear" in target_name
or "final_layer.adaLN_modulation" in source_name
):
logger.info(
f"Loading norm_out.linear: {source_name} -> {target_name}, "
f"shape: {tensor.shape}"
)
loaded_tensor = tensor.to(device=param.device, dtype=param.dtype)
param.data.copy_(loaded_tensor)
missing_keys.discard(target_name)
loaded_count += 1
del tensor, loaded_tensor
else:
# Debug logging for unmapped parameters
if "norm_out.linear" in target_name:
logger.warning(
f"norm_out.linear parameter {target_name} not found in model state_dict. "
f"Source: {source_name}"
)
unexpected_keys.append(target_name)
optional_missing_keys = []
required_missing_keys = []
for key in missing_keys:
if key.endswith(".bias"):
# Check if corresponding weight exists (if weight exists but bias doesn't, it's optional)
weight_key = key.replace(".bias", ".weight")
if weight_key not in missing_keys:
optional_missing_keys.append(key)
else:
required_missing_keys.append(key)
else:
required_missing_keys.append(key)
if required_missing_keys:
logger.warning(
f"Required missing keys (first 10): {required_missing_keys[:10]}..."
)
if optional_missing_keys:
logger.info(
f"Optional missing keys (bias parameters, {len(optional_missing_keys)} total): "
f"These will use default values (zeros)"
)
if unexpected_keys:
logger.warning(f"Unexpected keys (first 10): {unexpected_keys[:10]}...")
logger.info(f"Successfully loaded {loaded_count} weight tensors")
return model, reverse_param_names_mapping
def _convert_comfyui_weights(
self,
weight_iterator: Generator[tuple[str, torch.Tensor], None, None],
qkv_size: int,
mlp_hidden_dim: int,
has_guidance_embeds: bool,
) -> Generator[tuple[str, torch.Tensor], None, None]:
"""
Convert ComfyUI Flux weights to SGLang format.
Splits fused qkv weights into to_q/to_k/to_v plus proj_mlp.
Filters out guidance_in weights if model doesn't support guidance embeddings.
Handles scale/shift order difference between ComfyUI and AdaLayerNormContinuous.
"""
for name, tensor in weight_iterator:
if not has_guidance_embeds and name.startswith("guidance_in."):
logger.debug(
f"Skipping {name} (model doesn't support guidance embeddings)"
)
continue
# Split fused qkv in double blocks into separate q/k/v projections
match = re.match(
r"double_blocks\.(\d+)\.(img_attn|txt_attn)\.qkv\.(weight|bias)$", name
)
if match:
block_idx, attn_type, param_type = match.groups()
hidden_size = qkv_size // 3
if tensor.shape[0] < 3 * hidden_size:
logger.warning(
f"{name} shape {tensor.shape} smaller than expected qkv size {3 * hidden_size}, skipping"
)
continue
if param_type == "bias":
q_tensor = tensor[:hidden_size]
k_tensor = tensor[hidden_size : 2 * hidden_size]
v_tensor = tensor[2 * hidden_size : 3 * hidden_size]
else:
q_tensor = tensor[:hidden_size, :]
k_tensor = tensor[hidden_size : 2 * hidden_size, :]
v_tensor = tensor[2 * hidden_size : 3 * hidden_size, :]
target_prefix = f"transformer_blocks.{block_idx}.attn"
if attn_type == "img_attn":
yield f"{target_prefix}.to_q.{param_type}", q_tensor
yield f"{target_prefix}.to_k.{param_type}", k_tensor
yield f"{target_prefix}.to_v.{param_type}", v_tensor
else:
# txt_attn corresponds to encoder projections
yield f"{target_prefix}.add_q_proj.{param_type}", q_tensor
yield f"{target_prefix}.add_k_proj.{param_type}", k_tensor
yield f"{target_prefix}.add_v_proj.{param_type}", v_tensor
continue
match = re.match(r"single_blocks\.(\d+)\.linear1\.(weight|bias)$", name)
if match:
block_idx, param_type = match.groups()
expected_size = qkv_size + mlp_hidden_dim
if tensor.shape[0] < expected_size:
logger.warning(
f"linear1.{param_type} shape {tensor.shape} doesn't match "
f"expected size {expected_size}, skipping"
)
continue
# Split tensor
qkv_tensor = (
tensor[:qkv_size] if param_type == "bias" else tensor[:qkv_size, :]
)
mlp_tensor = (
tensor[qkv_size:] if param_type == "bias" else tensor[qkv_size:, :]
)
# Split qkv into q/k/v for single blocks
hidden_size = qkv_size // 3
if param_type == "bias":
q_tensor = qkv_tensor[:hidden_size]
k_tensor = qkv_tensor[hidden_size : 2 * hidden_size]
v_tensor = qkv_tensor[2 * hidden_size : 3 * hidden_size]
else:
q_tensor = qkv_tensor[:hidden_size, :]
k_tensor = qkv_tensor[hidden_size : 2 * hidden_size, :]
v_tensor = qkv_tensor[2 * hidden_size : 3 * hidden_size, :]
yield (
f"single_transformer_blocks.{block_idx}.attn.to_q.{param_type}",
q_tensor,
)
yield (
f"single_transformer_blocks.{block_idx}.attn.to_k.{param_type}",
k_tensor,
)
yield (
f"single_transformer_blocks.{block_idx}.attn.to_v.{param_type}",
v_tensor,
)
yield (
f"single_transformer_blocks.{block_idx}.proj_mlp.{param_type}",
mlp_tensor,
)
elif name == "final_layer.adaLN_modulation.1.weight":
# ComfyUI: output order is [shift, scale]
# AdaLayerNormContinuous: expects [scale, shift]
# Need to swap the first half and second half of the weight matrix
# Weight shape: (2 * hidden_size, hidden_size)
# Split into two halves and swap them
half_size = tensor.shape[0] // 2
shift_weights = tensor[:half_size, :]
scale_weights = tensor[half_size:, :]
# Swap: put scale first, then shift
swapped_tensor = torch.cat([scale_weights, shift_weights], dim=0)
logger.info(
f"Swapped scale/shift order for {name}: "
f"shape {tensor.shape} -> {swapped_tensor.shape}"
)
yield name, swapped_tensor
elif name == "final_layer.adaLN_modulation.1.bias":
# Same swap for bias: (2 * hidden_size,)
half_size = tensor.shape[0] // 2
shift_bias = tensor[:half_size]
scale_bias = tensor[half_size:]
swapped_tensor = torch.cat([scale_bias, shift_bias], dim=0)
logger.info(
f"Swapped scale/shift order for {name}: "
f"shape {tensor.shape} -> {swapped_tensor.shape}"
)
yield name, swapped_tensor
else:
# Other weights pass through (handled by param_names_mapping)
yield name, tensor
def _load_transformer_from_safetensors(
self,
server_args: ServerArgs,
loaded_modules: dict[str, torch.nn.Module] | None = None,
) -> dict[str, Any]:
"""
Load transformer directly from safetensors file without model_index.json.
"""
if loaded_modules is not None and "transformer" in loaded_modules:
logger.info("Using provided transformer module")
components = {
"transformer": loaded_modules["transformer"],
"scheduler": self.modules.get("scheduler"),
}
return components
if hasattr(server_args.pipeline_config, "dit_config"):
dit_config = server_args.pipeline_config.dit_config
if not isinstance(dit_config, FluxConfig):
logger.warning("dit_config is not FluxConfig, creating new FluxConfig")
dit_config = FluxConfig()
server_args.pipeline_config.dit_config = dit_config
else:
logger.info("Creating default FluxConfig")
dit_config = FluxConfig()
server_args.pipeline_config.dit_config = dit_config
# Set guidance_embeds to True for ComfyUI Flux models
dit_config.arch_config.guidance_embeds = True
logger.info("Set guidance_embeds=True for ComfyUI Flux model")
if dit_config.arch_config.param_names_mapping is None:
dit_config.arch_config.param_names_mapping = {}
# ComfyUI Flux uses different parameter names than SGLang Flux
# Key differences:
# - ComfyUI: single_blocks.{i}.linear1 (fused QKV + MLP input)
# - SGLang: single_transformer_blocks.{i}.attn.to_qkv + proj_mlp (separate)
# - ComfyUI: single_blocks.{i}.linear2
# - SGLang: single_transformer_blocks.{i}.proj_out
# - ComfyUI: double_blocks.{i}.img_attn.qkv / txt_attn.qkv
# - SGLang: transformer_blocks.{i}.attn.to_qkv / attn.to_added_qkv
# Note: For fused layers like linear1, we need custom weight splitting logic
# which will be handled in the weight conversion function below
comfyui_flux_mappings = {
# Double stream blocks - attention layers
r"double_blocks\.(\d+)\.img_attn\.qkv\.(weight|bias)$": (
r"transformer_blocks.\1.attn.to_qkv.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_attn\.qkv\.(weight|bias)$": (
r"transformer_blocks.\1.attn.to_added_qkv.\2",
None,
None,
),
r"double_blocks\.(\d+)\.img_attn\.proj\.(weight|bias)$": (
r"transformer_blocks.\1.attn.to_out.0.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_attn\.proj\.(weight|bias)$": (
r"transformer_blocks.\1.attn.to_add_out.\2",
None,
None,
),
r"double_blocks\.(\d+)\.img_attn\.norm\.query_norm\.scale$": (
r"transformer_blocks.\1.attn.norm_q.weight",
None,
None,
),
r"double_blocks\.(\d+)\.img_attn\.norm\.key_norm\.scale$": (
r"transformer_blocks.\1.attn.norm_k.weight",
None,
None,
),
r"double_blocks\.(\d+)\.txt_attn\.norm\.query_norm\.scale$": (
r"transformer_blocks.\1.attn.norm_added_q.weight",
None,
None,
),
r"double_blocks\.(\d+)\.txt_attn\.norm\.key_norm\.scale$": (
r"transformer_blocks.\1.attn.norm_added_k.weight",
None,
None,
),
# Double stream blocks - MLP layers (map to net structure)
r"double_blocks\.(\d+)\.img_mlp\.0\.(weight|bias)$": (
r"transformer_blocks.\1.ff.net.0.proj.\2",
None,
None,
),
r"double_blocks\.(\d+)\.img_mlp\.2\.(weight|bias)$": (
r"transformer_blocks.\1.ff.net.2.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_mlp\.0\.(weight|bias)$": (
r"transformer_blocks.\1.ff_context.net.0.proj.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_mlp\.2\.(weight|bias)$": (
r"transformer_blocks.\1.ff_context.net.2.\2",
None,
None,
),
# Double stream blocks - modulation layers
r"double_blocks\.(\d+)\.img_mod\.lin\.(weight|bias)$": (
r"transformer_blocks.\1.norm1.linear.\2",
None,
None,
),
r"double_blocks\.(\d+)\.txt_mod\.lin\.(weight|bias)$": (
r"transformer_blocks.\1.norm1_context.linear.\2",
None,
None,
),
# Single stream blocks - linear2 maps to proj_out
r"single_blocks\.(\d+)\.linear2\.(weight|bias)$": (
r"single_transformer_blocks.\1.proj_out.\2",
None,
None,
),
# Single stream blocks - norm layers (scale -> weight)
r"single_blocks\.(\d+)\.norm\.query_norm\.scale$": (
r"single_transformer_blocks.\1.attn.norm_q.weight",
None,
None,
),
r"single_blocks\.(\d+)\.norm\.key_norm\.scale$": (
r"single_transformer_blocks.\1.attn.norm_k.weight",
None,
None,
),
# Single stream blocks - modulation (maps to norm.linear)
r"single_blocks\.(\d+)\.modulation\.lin\.(weight|bias)$": (
r"single_transformer_blocks.\1.norm.linear.\2",
None,
None,
),
# Time and guidance embeddings
r"^time_in\.in_layer\.(weight|bias)$": (
r"time_text_embed.timestep_embedder.linear_1.\1",
None,
None,
),
r"^time_in\.out_layer\.(weight|bias)$": (
r"time_text_embed.timestep_embedder.linear_2.\1",
None,
None,
),
r"^txt_in\.(weight|bias)$": (r"context_embedder.\1", None, None),
r"^vector_in\.in_layer\.(weight|bias)$": (
r"time_text_embed.text_embedder.linear_1.\1",
None,
None,
),
r"^vector_in\.out_layer\.(weight|bias)$": (
r"time_text_embed.text_embedder.linear_2.\1",
None,
None,
),
# Final layer mappings
r"^final_layer\.linear\.(weight|bias)$": (r"proj_out.\1", None, None),
r"^final_layer\.norm_final\.(weight|bias)$": (r"norm_out.\1", None, None),
r"^final_layer\.adaLN_modulation\.1\.(weight|bias)$": (
r"norm_out.linear.\1",
None,
None,
),
# Image input embedding
r"^img_in\.(weight|bias)$": (r"x_embedder.\1", None, None),
# Guidance embeddings (if model supports guidance)
r"^guidance_in\.in_layer\.(weight|bias)$": (
r"time_text_embed.guidance_embedder.linear_1.\1",
None,
None,
),
r"^guidance_in\.out_layer\.(weight|bias)$": (
r"time_text_embed.guidance_embedder.linear_2.\1",
None,
None,
),
}
# Merge ComfyUI mappings with existing mappings (ComfyUI mappings take precedence)
updated_mapping = {
**dit_config.arch_config.param_names_mapping,
**comfyui_flux_mappings,
}
dit_config.arch_config.param_names_mapping = updated_mapping
logger.info(
"Added ComfyUI weight name mappings for Flux model. "
f"Total mappings: {len(updated_mapping)}"
)
cls_name = "FluxTransformer2DModel"
model_cls, _ = ModelRegistry.resolve_model_cls(cls_name)
logger.info("Resolved transformer class: %s", cls_name)
original_mapping = None
if comfyui_flux_mappings:
original_mapping = model_cls.param_names_mapping
model_cls.param_names_mapping = updated_mapping
logger.info(
"Temporarily updated model class param_names_mapping with ComfyUI mappings. "
f"Total mappings: {len(updated_mapping)}"
)
safetensors_list = [self.model_path]
logger.info("Loading weights from: %s", safetensors_list)
default_dtype = resolve_precision(
server_args, "dit", precision_attr="dit_precision"
)
server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "."
hf_config = {}
hidden_size = (
dit_config.arch_config.num_attention_heads
* dit_config.arch_config.attention_head_dim
)
mlp_ratio = getattr(dit_config.arch_config, "mlp_ratio", 4.0)
mlp_hidden_dim = int(hidden_size * mlp_ratio)
qkv_size = 3 * hidden_size
has_guidance_embeds = True
# Load and convert weights from safetensors file
model, reverse_param_names_mapping = (
self._load_and_convert_weights_from_safetensors(
model_cls=model_cls,
dit_config=dit_config,
hf_config=hf_config,
safetensors_list=safetensors_list,
updated_mapping=updated_mapping,
qkv_size=qkv_size,
mlp_hidden_dim=mlp_hidden_dim,
has_guidance_embeds=has_guidance_embeds,
default_dtype=default_dtype,
)
)
model = model.eval()
for param in model.parameters():
param.requires_grad = False
model.reverse_param_names_mapping = reverse_param_names_mapping
if original_mapping is not None:
model_cls.param_names_mapping = original_mapping
total_params = sum(p.numel() for p in model.parameters())
logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9)
components = {
"transformer": model,
"scheduler": self.modules.get("scheduler"),
}
logger.info("Successfully loaded modules: %s", list(components.keys()))
return components
def create_pipeline_stages(self, server_args: ServerArgs):
logger.info(
"ComfyUIFluxPipeline.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage"
)
self.add_stages(
[
ComfyUILatentPreparationStage(
scheduler=self.get_module("scheduler"),
transformer=self.get_module("transformer"),
),
DenoisingStage(
transformer=self.get_module("transformer"),
scheduler=self.get_module("scheduler"),
),
]
)
logger.info(
f"ComfyUIFluxPipeline stages created: {list(self._stage_name_mapping.keys())}"
)
EntryClass = ComfyUIFluxPipeline
@@ -1,359 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
import os
from itertools import chain
from typing import Any
import torch
from torch.distributed import init_device_mesh
from torch.distributed.fsdp import MixedPrecisionPolicy
from sglang.multimodal_gen.configs.models.dits.qwenimage import QwenImageDitConfig
from sglang.multimodal_gen.runtime.distributed import get_local_torch_device
from sglang.multimodal_gen.runtime.loader.fsdp_load import (
load_model_from_full_model_state_dict,
shard_model,
)
from sglang.multimodal_gen.runtime.loader.utils import (
get_param_names_mapping,
set_default_torch_dtype,
)
from sglang.multimodal_gen.runtime.loader.weight_utils import (
safetensors_weights_iterator,
)
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import (
ComfyUIPassThroughScheduler,
)
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
ComposedPipelineBase,
)
from sglang.multimodal_gen.runtime.pipelines_core.stages import (
ComfyUILatentPreparationStage,
DenoisingStage,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
from sglang.multimodal_gen.runtime.utils.precision import (
resolve_precision,
set_mixed_precision_policy,
)
logger = init_logger(__name__)
class ComfyUIQwenImagePipelineBase(LoRAPipeline, ComposedPipelineBase):
"""
Base pipeline for ComfyUI QwenImage integration with only denoising stage.
This pipeline requires pre-processed inputs:
- prompt_embeds: Pre-encoded text embeddings (list of tensors)
- latents: Pre-processed image latents in sequence format [B, S, D]
Usage:
generator = DiffGenerator.from_pretrained(
model_path="path/to/model",
pipeline_class_name="ComfyUIQwenImagePipeline",
device="cuda",
)
"""
# Subclasses should override this
zero_cond_t: bool = False
pipeline_name = "ComfyUIQwenImagePipeline"
_required_config_modules = [
"transformer",
"scheduler",
]
def initialize_pipeline(self, server_args: ServerArgs):
"""
Initialize the pipeline with ComfyUI pass-through scheduler.
This scheduler does not modify latents, allowing ComfyUI to handle denoising.
"""
self.modules["scheduler"] = ComfyUIPassThroughScheduler(
num_train_timesteps=1000
)
# Ensure VAE config is properly initialized even though we don't load the VAE model
vae_config = server_args.pipeline_config.vae_config
vae_config.post_init()
logger.info(
"Called vae_config.post_init() to set vae_scale_factor. "
f"vae_scale_factor={vae_config.arch_config.vae_scale_factor}"
)
def load_modules(
self,
server_args: ServerArgs,
loaded_modules: dict[str, torch.nn.Module] | None = None,
) -> dict[str, Any]:
"""
Load modules for ComfyUIQwenImagePipeline.
If model_path is a safetensors file, load transformer directly from it
without requiring model_index.json. Otherwise, fall back to default loading.
"""
if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"):
logger.info(
"Detected safetensors file, loading transformer directly from: %s",
self.model_path,
)
return self._load_transformer_from_safetensors(server_args, loaded_modules)
else:
logger.info(
"Model path is a directory, using default loading method: %s",
self.model_path,
)
return super().load_modules(server_args, loaded_modules)
def _load_transformer_from_safetensors(
self,
server_args: ServerArgs,
loaded_modules: dict[str, torch.nn.Module] | None = None,
) -> dict[str, Any]:
"""Load transformer directly from safetensors without model_index.json."""
# 1) Fast path: use provided module
if loaded_modules is not None and "transformer" in loaded_modules:
logger.info("Using provided transformer module")
return {
"transformer": loaded_modules["transformer"],
"scheduler": self.modules.get("scheduler"),
}
# 2) Build config and mappings
dit_config, updated_mapping, model_cls, default_dtype = (
self._prepare_dit_config_and_mapping(server_args)
)
safetensors_list = [self.model_path]
logger.info("Loading weights from: %s", safetensors_list)
# 3) Instantiate model (meta) and optionally shard
model = self._instantiate_model(
model_cls, dit_config, default_dtype, updated_mapping, server_args
)
# 4) Load weights
self._load_weights_into_model(
model, safetensors_list, default_dtype, updated_mapping, server_args
)
components = {
"transformer": model,
"scheduler": self.modules.get("scheduler"),
}
logger.info("Successfully loaded modules: %s", list(components.keys()))
return components
def _prepare_dit_config_and_mapping(self, server_args: ServerArgs):
from sglang.multimodal_gen.configs.models.dits.qwenimage import (
QwenImageArchConfig,
)
comfyui_arch_config = QwenImageArchConfig(
patch_size=2,
in_channels=64,
out_channels=16,
num_layers=60,
attention_head_dim=128,
num_attention_heads=24,
joint_attention_dim=3584,
pooled_projection_dim=768,
guidance_embeds=False,
axes_dims_rope=(16, 56, 56),
zero_cond_t=self.zero_cond_t,
)
dit_config = QwenImageDitConfig(arch_config=comfyui_arch_config)
server_args.pipeline_config.dit_config = dit_config
if dit_config.arch_config.param_names_mapping is None:
dit_config.arch_config.param_names_mapping = {}
comfyui_qwen_mappings = {r"^model\.diffusion_model\.(.*)$": r"\1"}
updated_mapping = {
**dit_config.arch_config.param_names_mapping,
**comfyui_qwen_mappings,
}
dit_config.arch_config.param_names_mapping = updated_mapping
logger.info(
"Added ComfyUI weight name mappings to param_names_mapping. "
f"Total mappings: {len(updated_mapping)}"
)
cls_name = "QwenImageTransformer2DModel"
model_cls, _ = ModelRegistry.resolve_model_cls(cls_name)
logger.info("Resolved transformer class: %s", cls_name)
default_dtype = resolve_precision(
server_args, "dit", precision_attr="dit_precision"
)
server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "."
assert server_args.hsdp_shard_dim is not None, "hsdp_shard_dim must be set"
logger.info(
"Loading %s from safetensors file, default_dtype: %s",
cls_name,
default_dtype,
)
return dit_config, updated_mapping, model_cls, default_dtype
def _instantiate_model(
self,
model_cls,
dit_config,
default_dtype,
updated_mapping,
server_args: ServerArgs,
):
from sglang.multimodal_gen.runtime.platforms import current_platform
hf_config = {}
original_mapping = model_cls.param_names_mapping
model_cls.param_names_mapping = updated_mapping
logger.info(
"Temporarily updated model class param_names_mapping with ComfyUI mappings. "
f"Total mappings: {len(updated_mapping)}"
)
try:
# precision-constraint: FSDP mixed precision currently uses bf16
# parameters and fp32 reduction regardless of model load dtype.
mp_policy = MixedPrecisionPolicy(
torch.bfloat16, torch.float32, None, cast_forward_inputs=False
)
set_mixed_precision_policy(
param_dtype=torch.bfloat16,
reduce_dtype=torch.float32,
output_dtype=None,
mp_policy=mp_policy,
)
with set_default_torch_dtype(default_dtype), torch.device("meta"):
model = model_cls(**{"config": dit_config, "hf_config": hf_config})
use_fsdp = server_args.should_use_fsdp_for_component("transformer")
if current_platform.is_mps():
use_fsdp = False
logger.info("Disabling FSDP for MPS platform as it's not compatible")
if use_fsdp:
device_mesh = init_device_mesh(
current_platform.device_type,
mesh_shape=(
server_args.hsdp_replicate_dim,
server_args.hsdp_shard_dim,
),
mesh_dim_names=("replicate", "shard"),
)
shard_model(
model,
cpu_offload=False,
reshard_after_forward=True,
mp_policy=mp_policy,
mesh=device_mesh,
fsdp_shard_conditions=getattr(
model, "_fsdp_shard_conditions", None
),
pin_cpu_memory=server_args.pin_cpu_memory,
)
finally:
model_cls.param_names_mapping = original_mapping
return model
def _load_weights_into_model(
self,
model,
safetensors_list,
default_dtype,
updated_mapping,
server_args: ServerArgs,
):
use_fsdp = server_args.should_use_fsdp_for_component("transformer")
component_starts_on_cpu = server_args.should_start_component_on_cpu(
"transformer"
)
# Create weight iterator for loading
weight_iterator = safetensors_weights_iterator(safetensors_list)
# Load weights
param_names_mapping_fn = get_param_names_mapping(updated_mapping)
load_model_from_full_model_state_dict(
model,
weight_iterator,
get_local_torch_device(),
default_dtype,
strict=True,
cpu_offload=component_starts_on_cpu and not use_fsdp,
param_names_mapping=param_names_mapping_fn,
)
# Check for meta parameters
for n, p in chain(model.named_parameters(), model.named_buffers()):
if p.is_meta:
raise RuntimeError(f"Unexpected param or buffer {n} on meta device.")
if isinstance(p, torch.nn.Parameter):
p.requires_grad = False
total_params = sum(p.numel() for p in model.parameters())
logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9)
def create_pipeline_stages(self, server_args: ServerArgs):
logger.info(
f"{self.__class__.__name__}.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage"
)
self.add_stages(
[
ComfyUILatentPreparationStage(
scheduler=self.get_module("scheduler"),
transformer=self.get_module("transformer"),
),
DenoisingStage(
transformer=self.get_module("transformer"),
scheduler=self.get_module("scheduler"),
),
]
)
logger.info(
f"{self.__class__.__name__} stages created: {list(self._stage_name_mapping.keys())}"
)
class ComfyUIQwenImagePipeline(ComfyUIQwenImagePipelineBase):
"""ComfyUI QwenImage pipeline for text-to-image generation."""
pipeline_name = "ComfyUIQwenImagePipeline"
zero_cond_t = False
from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import (
QwenImagePipelineConfig,
)
from sglang.multimodal_gen.configs.sample.qwenimage import QwenImageSamplingParams
pipeline_config_cls = QwenImagePipelineConfig
sampling_params_cls = QwenImageSamplingParams
class ComfyUIQwenImageEditPipeline(ComfyUIQwenImagePipelineBase):
"""ComfyUI QwenImage pipeline for image-to-image editing."""
pipeline_name = "ComfyUIQwenImageEditPipeline"
zero_cond_t = True
from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import (
QwenImageEditPlusPipelineConfig,
)
from sglang.multimodal_gen.configs.sample.qwenimage import (
QwenImageEditPlusSamplingParams,
)
pipeline_config_cls = QwenImageEditPlusPipelineConfig
sampling_params_cls = QwenImageEditPlusSamplingParams
EntryClass = [ComfyUIQwenImagePipeline, ComfyUIQwenImageEditPipeline]
@@ -1,412 +0,0 @@
# Copied and adapted from: https://github.com/hao-ai-lab/FastVideo
# SPDX-License-Identifier: Apache-2.0
import os
import re
from collections.abc import Generator
from itertools import chain
from typing import Any
import torch
from torch.distributed import init_device_mesh
from torch.distributed.fsdp import MixedPrecisionPolicy
from sglang.multimodal_gen.configs.models.dits.zimage import ZImageDitConfig
from sglang.multimodal_gen.runtime.distributed import get_local_torch_device
from sglang.multimodal_gen.runtime.loader.fsdp_load import (
load_model_from_full_model_state_dict,
shard_model,
)
from sglang.multimodal_gen.runtime.loader.utils import (
get_param_names_mapping,
set_default_torch_dtype,
)
from sglang.multimodal_gen.runtime.loader.weight_utils import (
safetensors_weights_iterator,
)
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import (
ComfyUIPassThroughScheduler,
)
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
ComposedPipelineBase,
)
from sglang.multimodal_gen.runtime.pipelines_core.stages import (
ComfyUILatentPreparationStage,
DenoisingStage,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
from sglang.multimodal_gen.runtime.utils.precision import (
resolve_precision,
set_mixed_precision_policy,
)
logger = init_logger(__name__)
class ComfyUIZImagePipeline(LoRAPipeline, ComposedPipelineBase):
"""
Simplified pipeline for ComfyUI integration with only denoising stage.
This pipeline requires pre-processed inputs:
- prompt_embeds: Pre-encoded text embeddings (list of tensors)
- negative_prompt_embeds: Pre-encoded negative prompt embeddings (if using CFG)
- latents: Optional initial noise latents (will be generated if not provided)
Usage:
generator = DiffGenerator.from_pretrained(
model_path="path/to/model",
pipeline_class_name="ComfyUIZImagePipeline",
device="cuda",
)
"""
pipeline_name = "ComfyUIZImagePipeline"
from sglang.multimodal_gen.configs.pipeline_configs.zimage import (
ZImagePipelineConfig,
)
from sglang.multimodal_gen.configs.sample.zimage import ZImageSamplingParams
pipeline_config_cls = ZImagePipelineConfig
sampling_params_cls = ZImageSamplingParams
_required_config_modules = [
"transformer",
"scheduler",
]
def initialize_pipeline(self, server_args: ServerArgs):
"""
Initialize the pipeline with ComfyUI pass-through scheduler.
This scheduler does not modify latents, allowing ComfyUI to handle denoising.
"""
self.modules["scheduler"] = ComfyUIPassThroughScheduler(
num_train_timesteps=1000
)
# Ensure VAE config is properly initialized even though we don't load the VAE model
# This is necessary because get_freqs_cis uses spatial_compression_ratio
if hasattr(server_args.pipeline_config, "vae_config"):
vae_config = server_args.pipeline_config.vae_config
if hasattr(vae_config, "post_init") and not hasattr(
vae_config, "_post_init_called"
):
vae_config.post_init()
logger.info(
"Called vae_config.post_init() to set spatial_compression_ratio. "
f"spatial_compression_ratio={vae_config.arch_config.spatial_compression_ratio}"
)
def load_modules(
self,
server_args: ServerArgs,
loaded_modules: dict[str, torch.nn.Module] | None = None,
) -> dict[str, Any]:
"""
Load modules for ComfyUIZImagePipeline.
If model_path is a safetensors file, load transformer directly from it
without requiring model_index.json. Otherwise, fall back to default loading.
"""
if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"):
logger.info(
"Detected safetensors file, loading transformer directly from: %s",
self.model_path,
)
return self._load_transformer_from_safetensors(server_args, loaded_modules)
else:
logger.info(
"Model path is a directory, using default loading method: %s",
self.model_path,
)
return super().load_modules(server_args, loaded_modules)
def _convert_comfyui_qkv_weights(
self,
weight_iterator: Generator[tuple[str, torch.Tensor], None, None],
dim: int,
num_heads: int,
num_kv_heads: int,
) -> Generator[tuple[str, torch.Tensor], None, None]:
"""
Convert ComfyUI zimage qkv weights to SGLang format.
Splits merged qkv.weight into separate to_q, to_k, to_v weights.
Args:
weight_iterator: Iterator yielding (name, tensor) pairs from safetensors
dim: Model dimension
num_heads: Number of attention heads
num_kv_heads: Number of key-value heads
Yields:
(name, tensor) pairs with qkv weights split into to_q, to_k, to_v
"""
head_dim = dim // num_heads
q_size = dim
k_size = head_dim * num_kv_heads
for name, tensor in weight_iterator:
# Match qkv weights in layers, noise_refiner, or context_refiner
# Pattern: (layers|noise_refiner|context_refiner).{i}.attention.qkv.(weight|bias)
match = re.match(
r"(layers|noise_refiner|context_refiner)\.(\d+)\.attention\.qkv\.(weight|bias)$",
name,
)
if match:
module_name, layer_idx, param_type = match.groups()
base_name = f"{module_name}.{layer_idx}.attention"
if param_type == "weight":
# Weight shape: (q_size + k_size + v_size, dim)
# Split into q, k, v
q_weight = tensor[:q_size, :]
k_weight = tensor[q_size : q_size + k_size, :]
v_weight = tensor[q_size + k_size :, :]
logger.debug(
f"Splitting {name} (shape {tensor.shape}) into "
f"to_q ({q_weight.shape}), to_k ({k_weight.shape}), to_v ({v_weight.shape})"
)
yield f"{base_name}.to_q.weight", q_weight
yield f"{base_name}.to_k.weight", k_weight
yield f"{base_name}.to_v.weight", v_weight
else: # bias
# Bias shape: (q_size + k_size + v_size,)
# Split into q, k, v
q_bias = tensor[:q_size]
k_bias = tensor[q_size : q_size + k_size]
v_bias = tensor[q_size + k_size :]
logger.debug(
f"Splitting {name} (shape {tensor.shape}) into "
f"to_q ({q_bias.shape}), to_k ({k_bias.shape}), to_v ({v_bias.shape})"
)
yield f"{base_name}.to_q.bias", q_bias
yield f"{base_name}.to_k.bias", k_bias
yield f"{base_name}.to_v.bias", v_bias
else:
# Pass through other weights unchanged
yield name, tensor
def _load_transformer_from_safetensors(
self,
server_args: ServerArgs,
loaded_modules: dict[str, torch.nn.Module] | None = None,
) -> dict[str, Any]:
"""
Load transformer directly from safetensors file without model_index.json.
This method:
1. Uses hardcoded ZImageDitConfig for zimage model
2. Loads transformer from the safetensors file
3. Uses ComfyUIPassThroughScheduler (already created in initialize_pipeline)
"""
# Check if transformer is already provided
if loaded_modules is not None and "transformer" in loaded_modules:
logger.info("Using provided transformer module")
components = {
"transformer": loaded_modules["transformer"],
"scheduler": self.modules.get("scheduler"),
}
return components
if hasattr(server_args.pipeline_config, "dit_config"):
dit_config = server_args.pipeline_config.dit_config
if not isinstance(dit_config, ZImageDitConfig):
logger.warning(
"dit_config is not ZImageDitConfig, creating new ZImageDitConfig"
)
dit_config = ZImageDitConfig()
server_args.pipeline_config.dit_config = dit_config
else:
logger.info("Creating default ZImageDitConfig")
dit_config = ZImageDitConfig()
server_args.pipeline_config.dit_config = dit_config
if dit_config.arch_config.param_names_mapping is None:
dit_config.arch_config.param_names_mapping = {}
# Add mappings for norm layers: map from ComfyUI format (k_norm/q_norm) to SGLang format (norm_k/norm_q)
# The regex matches the source name from safetensors, and the tuple specifies the target name in the model
# Note: qkv weights are handled separately by _convert_comfyui_qkv_weights function
comfyui_norm_mappings = {
r"(.*)\.attention\.k_norm\.weight$": (
r"\1.attention.norm_k.weight",
None,
None,
),
r"(.*)\.attention\.q_norm\.weight$": (
r"\1.attention.norm_q.weight",
None,
None,
),
r"(.*)\.attention\.out\.weight$": (
r"\1.attention.to_out.0.weight",
None,
None,
),
r"^final_layer\.(.*)$": (r"all_final_layer.2-1.\1", None, None),
r"^x_embedder\.(.*)$": (r"all_x_embedder.2-1.\1", None, None),
}
# Merge ComfyUI mappings with existing mappings (ComfyUI mappings take precedence)
updated_mapping = {
**dit_config.arch_config.param_names_mapping,
**comfyui_norm_mappings,
}
dit_config.arch_config.param_names_mapping = updated_mapping
logger.info(
"Added ComfyUI weight name mappings (k_norm/q_norm -> norm_k/norm_q) to param_names_mapping. "
f"Total mappings: {len(updated_mapping)}"
)
cls_name = "ZImageTransformer2DModel"
model_cls, _ = ModelRegistry.resolve_model_cls(cls_name)
logger.info("Resolved transformer class: %s", cls_name)
safetensors_list = [self.model_path]
logger.info("Loading weights from: %s", safetensors_list)
default_dtype = resolve_precision(
server_args, "dit", precision_attr="dit_precision"
)
server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "."
hf_config = {}
assert server_args.hsdp_shard_dim is not None, "hsdp_shard_dim must be set"
logger.info(
"Loading %s from safetensors file, default_dtype: %s",
cls_name,
default_dtype,
)
original_mapping = model_cls.param_names_mapping
model_cls.param_names_mapping = updated_mapping
logger.info(
"Temporarily updated model class param_names_mapping with ComfyUI mappings. "
f"Total mappings: {len(updated_mapping)}"
)
try:
# Create model first (same as maybe_load_fsdp_model)
from sglang.multimodal_gen.runtime.platforms import current_platform
# precision-constraint: FSDP mixed precision currently uses bf16
# parameters and fp32 reduction regardless of model load dtype.
mp_policy = MixedPrecisionPolicy(
torch.bfloat16, torch.float32, None, cast_forward_inputs=False
)
set_mixed_precision_policy(
param_dtype=torch.bfloat16,
reduce_dtype=torch.float32,
output_dtype=None,
mp_policy=mp_policy,
)
with set_default_torch_dtype(default_dtype), torch.device("meta"):
model = model_cls(**{"config": dit_config, "hf_config": hf_config})
# Check if we should use FSDP
use_fsdp = server_args.should_use_fsdp_for_component("transformer")
component_starts_on_cpu = server_args.should_start_component_on_cpu(
"transformer"
)
if current_platform.is_mps():
use_fsdp = False
logger.info("Disabling FSDP for MPS platform as it's not compatible")
if use_fsdp:
device_mesh = init_device_mesh(
current_platform.device_type,
mesh_shape=(
server_args.hsdp_replicate_dim,
server_args.hsdp_shard_dim,
),
mesh_dim_names=("replicate", "shard"),
)
shard_model(
model,
cpu_offload=False,
reshard_after_forward=True,
mp_policy=mp_policy,
mesh=device_mesh,
fsdp_shard_conditions=getattr(
model, "_fsdp_shard_conditions", None
),
pin_cpu_memory=server_args.pin_cpu_memory,
)
# Get model dimensions for qkv splitting
arch_config = dit_config.arch_config
dim = arch_config.dim
num_heads = arch_config.num_attention_heads
num_kv_heads = arch_config.n_kv_heads
# Create weight iterator with qkv conversion
base_weight_iterator = safetensors_weights_iterator(safetensors_list)
converted_weight_iterator = self._convert_comfyui_qkv_weights(
base_weight_iterator, dim, num_heads, num_kv_heads
)
# Load weights
param_names_mapping_fn = get_param_names_mapping(updated_mapping)
load_model_from_full_model_state_dict(
model,
converted_weight_iterator,
get_local_torch_device(),
default_dtype,
strict=True,
cpu_offload=component_starts_on_cpu and not use_fsdp,
param_names_mapping=param_names_mapping_fn,
)
# Check for meta parameters
for n, p in chain(model.named_parameters(), model.named_buffers()):
if p.is_meta:
raise RuntimeError(
f"Unexpected param or buffer {n} on meta device."
)
if isinstance(p, torch.nn.Parameter):
p.requires_grad = False
finally:
model_cls.param_names_mapping = original_mapping
total_params = sum(p.numel() for p in model.parameters())
logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9)
components = {
"transformer": model,
"scheduler": self.modules.get("scheduler"),
}
logger.info("Successfully loaded modules: %s", list(components.keys()))
return components
def create_pipeline_stages(self, server_args: ServerArgs):
logger.info(
"ComfyUIZImagePipeline.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage"
)
self.add_stages(
[
ComfyUILatentPreparationStage(
scheduler=self.get_module("scheduler"),
transformer=self.get_module("transformer"),
),
DenoisingStage(
transformer=self.get_module("transformer"),
scheduler=self.get_module("scheduler"),
),
]
)
logger.info(
f"ComfyUIZImagePipeline stages created: {list(self._stage_name_mapping.keys())}"
)
EntryClass = ComfyUIZImagePipeline
@@ -2,6 +2,8 @@
# SPDX-License-Identifier: Apache-2.0
from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig
from sglang.multimodal_gen.configs.sample.flux import FluxSamplingParams
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
ComposedPipelineBase,
@@ -32,6 +34,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs):
class FluxPipeline(LoRAPipeline, ComposedPipelineBase):
pipeline_name = "FluxPipeline"
# Used when the checkpoint is a single safetensors file with no
# model_index.json to derive these from.
pipeline_config_cls = FluxPipelineConfig
sampling_params_cls = FluxSamplingParams
_required_config_modules = [
"text_encoder",
"text_encoder_2",
@@ -190,6 +190,15 @@ class MiniMaxH3Pipeline(LoRAPipeline, ComposedPipelineBase):
)
)
def create_comfyui_stages(self, server_args: ServerArgs) -> None:
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import (
MiniMaxH3ComfyUIStepStage,
)
self.add_stage(
MiniMaxH3ComfyUIStepStage(transformer=self.get_module("transformer"))
)
class FastH3Pipeline(MiniMaxH3Pipeline):
"""FastH3: 4-step DMD2-distilled MiniMax-H3 (t2va only).
@@ -3,6 +3,14 @@
# SPDX-License-Identifier: Apache-2.0
from diffusers.image_processor import VaeImageProcessor
from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import (
QwenImageEditPlusPipelineConfig,
QwenImagePipelineConfig,
)
from sglang.multimodal_gen.configs.sample.qwenimage import (
QwenImageEditPlusSamplingParams,
QwenImageSamplingParams,
)
from sglang.multimodal_gen.runtime.disaggregation.roles import RoleType
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
@@ -39,6 +47,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs):
class QwenImagePipeline(LoRAPipeline, ComposedPipelineBase):
pipeline_name = "QwenImagePipeline"
# Used when the checkpoint is a single safetensors file with no
# model_index.json to derive these from.
pipeline_config_cls = QwenImagePipelineConfig
sampling_params_cls = QwenImageSamplingParams
_required_config_modules = [
"text_encoder",
"tokenizer",
@@ -84,6 +97,11 @@ class QwenImageEditPipeline(LoRAPipeline, ComposedPipelineBase):
class QwenImageEditPlusPipeline(QwenImageEditPipeline):
pipeline_name = "QwenImageEditPlusPipeline"
# Used when the checkpoint is a single safetensors file with no
# model_index.json to derive these from.
pipeline_config_cls = QwenImageEditPlusPipelineConfig
sampling_params_cls = QwenImageEditPlusSamplingParams
def prepare_mu_layered(batch: Req, server_args: ServerArgs):
base_seqlen = 256 * 256 / 16 / 16
@@ -1,6 +1,8 @@
# Copied and adapted from: https://github.com/hao-ai-lab/FastVideo
# SPDX-License-Identifier: Apache-2.0
from sglang.multimodal_gen.configs.pipeline_configs.zimage import ZImagePipelineConfig
from sglang.multimodal_gen.configs.sample.zimage import ZImageSamplingParams
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline, Req
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
ComposedPipelineBase,
@@ -27,6 +29,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs):
class ZImagePipeline(LoRAPipeline, ComposedPipelineBase):
pipeline_name = "ZImagePipeline"
# Used when the checkpoint is a single safetensors file with no
# model_index.json to derive these from.
pipeline_config_cls = ZImagePipelineConfig
sampling_params_cls = ZImageSamplingParams
_required_config_modules = [
"text_encoder",
"tokenizer",
@@ -0,0 +1,226 @@
# SPDX-License-Identifier: Apache-2.0
"""Worker-side support for ``--comfyui-mode``.
ComfyUI owns the sampler loop and calls SGLang once per DiT step. This
module covers both halves of that contract:
- pipeline assembly: drop every module except the transformer, install a
pass-through scheduler, keep only the stages needed for one forward
- run cache: after the first step, text embeddings and packed extras stay
on the worker; later steps send latents and the timestep
Single-file / GGUF DiT loading lives in
``sglang.multimodal_gen.runtime.loader.comfyui_checkpoints``.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Callable
from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import (
ComfyUIPassThroughScheduler,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
if TYPE_CHECKING:
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
ComposedPipelineBase,
)
COMFYUI_REQUIRED_MODULES = ["transformer", "scheduler"]
_CONDITIONING_FIELDS = (
"prompt_embeds",
"negative_prompt_embeds",
"prompt_seq_lens",
"negative_prompt_seq_lens",
"pooled_embeds",
"neg_pooled_embeds",
"image_latent",
"vae_image_sizes",
"prompt_attention_mask",
"negative_attention_mask",
"prompt_embeds_mask",
"negative_prompt_embeds_mask",
"sigmas",
)
_SESSION_SKIP_EXTRA = frozenset(
{"comfyui_session_id", "comfyui_cond_key", "comfyui_cache_fp"}
)
_SESSIONS: dict[str, dict[str, Any]] = {}
_RUNS: dict[str, Any] = {}
def is_comfyui_mode(server_args: ServerArgs) -> bool:
return bool(server_args.comfyui_mode)
def initialize_comfyui_pipeline(
pipeline: ComposedPipelineBase, server_args: ServerArgs
) -> None:
"""Install the pass-through scheduler and finish deriving VAE geometry.
The VAE model itself is never loaded, but its config still carries the
compression ratios that RoPE frequency construction reads.
"""
pipeline.modules["scheduler"] = ComfyUIPassThroughScheduler(
num_train_timesteps=1000
)
vae_config = getattr(server_args.pipeline_config, "vae_config", None)
if (
vae_config is not None
and hasattr(vae_config, "post_init")
and not hasattr(vae_config, "_post_init_called")
):
# Flux / Qwen RoPE reads arch_config.vae_scale_factor, which
# post_init() derives from dim_mult. Do not skip just because a
# ComfyUI DiT checkpoint has no latents_mean — that field is only
# required when this process owns the VAE (MiniMax-H3 native).
vae_config.post_init()
def create_comfyui_pipeline_stages(
pipeline: ComposedPipelineBase, server_args: ServerArgs
) -> None:
if hasattr(pipeline, "create_comfyui_stages"):
pipeline.create_comfyui_stages(server_args)
return
from sglang.multimodal_gen.runtime.pipelines_core.stages import (
ComfyUILatentPreparationStage,
DenoisingStage,
)
transformer = pipeline.get_module("transformer")
scheduler = pipeline.get_module("scheduler")
pipeline.add_stages(
[
ComfyUILatentPreparationStage(scheduler=scheduler, transformer=transformer),
DenoisingStage(transformer=transformer, scheduler=scheduler),
]
)
def session_id_from_req(req) -> str | None:
extra = getattr(req, "extra", None) or {}
sid = extra.get("comfyui_session_id")
return sid if sid else None
def _session_family(session_id: str) -> str:
return session_id.split(":", 1)[0]
def _evict_other_runs(session_id: str) -> None:
"""Drop older ``{uuid}:{run}`` slots for this executor."""
prefix = _session_family(session_id) + ":"
for store in (_SESSIONS, _RUNS):
for key in list(store):
if key.startswith(prefix) and key != session_id:
store.pop(key, None)
def bind_comfyui_session(req):
"""Restore cached conditioning, then refresh the cache from whatever is set.
Conditioning is stored per ``comfyui_cond_key`` so CFG pos/neg do not
overwrite each other. Packed extras are restored only when
``comfyui_cache_fp`` matches; a mismatch must not revive a stale layout.
"""
sid = session_id_from_req(req)
if not sid:
return req
_evict_other_runs(sid)
extra = dict(getattr(req, "extra", None) or {})
cond_key = extra.get("comfyui_cond_key") or "_"
cached = _SESSIONS.get(sid) or {}
cond_snap = (cached.get("_conds") or {}).get(cond_key) or {}
current_fp = extra.get("comfyui_cache_fp")
cached_fp = cached.get("_fp")
extra_ok = current_fp is None or cached_fp is None or current_fp == cached_fp
for name, value in cond_snap.items():
if _is_empty(getattr(req, name, None)):
setattr(req, name, value)
if extra_ok:
for key, value in (cached.get("_extra") or {}).items():
if _is_empty(extra.get(key)):
extra[key] = value
req.extra = extra
new_cond = {}
for name in _CONDITIONING_FIELDS:
value = getattr(req, name, None)
if not _is_empty(value):
new_cond[name] = value
extra = getattr(req, "extra", None) or {}
extra_snapshot = {}
for key, value in extra.items():
if key in _SESSION_SKIP_EXTRA or _is_empty(value):
continue
extra_snapshot[key] = value
conds = dict(cached.get("_conds") or {})
if new_cond:
conds[cond_key] = new_cond
snapshot: dict[str, Any] = {"_conds": conds}
if extra_snapshot:
snapshot["_extra"] = extra_snapshot
if current_fp is not None:
snapshot["_fp"] = current_fp
elif extra_ok and cached_fp is not None:
snapshot["_fp"] = cached_fp
if conds or extra_snapshot:
_SESSIONS[sid] = snapshot
return req
def get_run_state(req):
sid = session_id_from_req(req)
if sid:
return _RUNS.get(sid)
return None
def set_run_state(req, state: Any) -> None:
sid = session_id_from_req(req)
if sid:
_RUNS[sid] = state
def pop_run_state(req):
sid = session_id_from_req(req)
if sid:
return _RUNS.pop(sid, None)
return None
def get_or_create_run_state(req, factory: Callable[[], Any]):
"""Session-scoped opaque state. Created once per run."""
existing = get_run_state(req)
if existing is not None:
return existing
state = factory()
set_run_state(req, state)
return state
def release_run_state(session_id: str | None) -> None:
"""Drop per-run objects; keep cached conditioning for the next step."""
if session_id:
_RUNS.pop(session_id, None)
def release_comfyui_session(session_id: str | None) -> None:
if session_id:
_SESSIONS.pop(session_id, None)
_RUNS.pop(session_id, None)
def _is_empty(value) -> bool:
if value is None:
return True
if isinstance(value, (list, tuple)):
return len(value) == 0
return False
@@ -18,6 +18,10 @@ from sglang.multimodal_gen.runtime.disaggregation.roles import (
RoleType,
filter_modules_for_role,
)
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import (
is_comfyui_single_file,
load_comfyui_transformer,
)
from sglang.multimodal_gen.runtime.loader.component_loaders.component_loader import (
ComponentLoader,
PipelineComponentLoader,
@@ -32,6 +36,12 @@ from sglang.multimodal_gen.runtime.managers.memory_managers.component_manager im
ComponentResidencyStrategy,
get_global_component_residency_manager,
)
from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import (
COMFYUI_REQUIRED_MODULES,
create_comfyui_pipeline_stages,
initialize_comfyui_pipeline,
is_comfyui_mode,
)
from sglang.multimodal_gen.runtime.pipelines_core.executors.pipeline_executor import (
PipelineExecutor,
)
@@ -123,6 +133,9 @@ class ComposedPipelineBase(ABC):
)
if base_required_config_modules is None:
raise NotImplementedError("Subclass must set _required_config_modules")
if is_comfyui_mode(server_args):
# ComfyUI owns text encoding, VAE decode, and the sampler loop.
base_required_config_modules = COMFYUI_REQUIRED_MODULES
self._unfiltered_required_config_modules = tuple(base_required_config_modules)
self._required_config_modules = list(self._unfiltered_required_config_modules)
self._extra_config_module_map = dict(self._extra_config_module_map)
@@ -166,6 +179,12 @@ class ComposedPipelineBase(ABC):
def __post_init__(self) -> None:
assert self.server_args is not None, "server_args must be set"
if is_comfyui_mode(self.server_args):
initialize_comfyui_pipeline(self, self.server_args)
logger.info("Creating ComfyUI pipeline stages...")
create_comfyui_pipeline_stages(self, self.server_args)
return
self.initialize_pipeline(self.server_args)
logger.info("Creating pipeline stages...")
@@ -425,6 +444,8 @@ class ComposedPipelineBase(ABC):
loaded_modules: Optional[Dict[str, torch.nn.Module]] = None,
If provided, loaded_modules will be used instead of loading from config/pretrained weights.
"""
if is_comfyui_mode(server_args) and is_comfyui_single_file(self.model_path):
return load_comfyui_transformer(self, server_args, loaded_modules)
model_index = self._load_config()
logger.info("Loading pipeline modules from config: %s", model_index)
@@ -155,7 +155,7 @@ class Req:
raw_latent_shape: torch.Tensor | None = None
did_sp_shard_latents: bool = False
sp_video_start_frame: int = 0
noise_pred: torch.Tensor | None = None
noise_pred: torch.Tensor | list | tuple | None = None
# vae-encoded condition image
image_latent: torch.Tensor | list[torch.Tensor] | None = None
condition_image_latent_ids: torch.Tensor | list[torch.Tensor] | None = None
@@ -502,7 +502,7 @@ class OutputBatch:
metrics_list: Optional[list[Optional[RequestMetrics]]] = None
# For ComfyUI integration: noise prediction from denoising stage
noise_pred: torch.Tensor | None = None
noise_pred: torch.Tensor | list | tuple | None = None
peak_memory_mb: float = 0.0
usage: dict[str, Any] | None = None
@@ -1,114 +1,42 @@
# SPDX-License-Identifier: Apache-2.0
"""
ComfyUI latent preparation stage with device mismatch fix.
This stage extends LatentPreparationStage to handle device mismatch issues
that occur when tensors are pickled and unpickled via broadcast_pyobj in
multi-GPU scenarios.
"""ComfyUI latent prep: restore the worker session and bind the pass-through scheduler.
Multi-rank hops move CUDA tensors with NCCL, so this stage no longer walks
every field to fix pickle/gloo device mismatches.
"""
import dataclasses
import torch
from sglang.multimodal_gen.runtime.distributed import (
get_local_torch_device,
get_sp_group,
from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import (
bind_comfyui_session,
)
from sglang.multimodal_gen.runtime.pipelines_core.diffusion_scheduler_utils import (
get_or_create_request_scheduler,
)
from sglang.multimodal_gen.runtime.distributed.parallel_state import get_sp_world_size
from sglang.multimodal_gen.runtime.pipelines_core.stages.latent_preparation import (
LatentPreparationStage,
)
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
logger = init_logger(__name__)
class ComfyUILatentPreparationStage(LatentPreparationStage):
"""
ComfyUI-specific latent preparation stage with device mismatch fix.
"""One DiT step: restore cached conditioning, then prepare latents."""
This stage extends LatentPreparationStage to automatically fix device
mismatches for tensor fields on non-source ranks in multi-GPU scenarios.
"""
@staticmethod
def _fix_tensor_device(value, target_device):
"""Recursively fix tensor device, handling single tensors, lists, and tuples."""
if isinstance(value, torch.Tensor):
if value.device != target_device:
return value.detach().clone().to(target_device)
return value
elif isinstance(value, list):
return [
ComfyUILatentPreparationStage._fix_tensor_device(v, target_device)
for v in value
]
elif isinstance(value, tuple):
return tuple(
ComfyUILatentPreparationStage._fix_tensor_device(v, target_device)
for v in value
)
return value
@staticmethod
def _has_tensor(value):
"""Check if value contains any tensor."""
if isinstance(value, torch.Tensor):
return True
elif isinstance(value, (list, tuple)):
return any(ComfyUILatentPreparationStage._has_tensor(v) for v in value)
return False
def verify_input(self, batch, server_args):
bind_comfyui_session(batch)
return super().verify_input(batch, server_args)
def forward(self, batch, server_args):
"""
Prepare latents with device mismatch fix for ComfyUI pipelines.
This method first fixes device mismatches for all tensor fields,
then calls the parent class's forward method, and ensures raw_latent_shape
is set correctly (before packing, for proper unpadding later).
"""
# Fix device mismatch for tensor fields on non-source ranks
if get_sp_world_size() > 1:
sp_group = get_sp_group()
target_device = get_local_torch_device()
if sp_group.rank != 0:
logger.debug(
f"[ComfyUILatentPreparationStage] Fixing tensor device on rank={sp_group.rank} "
f"target_device={target_device}"
)
if dataclasses.is_dataclass(batch):
for field in dataclasses.fields(batch):
value = getattr(batch, field.name, None)
if value is not None and self._has_tensor(value):
fixed_value = self._fix_tensor_device(value, target_device)
setattr(batch, field.name, fixed_value)
else:
for attr_name in dir(batch):
if not attr_name.startswith("_") and not callable(
getattr(batch, attr_name, None)
):
try:
value = getattr(batch, attr_name, None)
if value is not None and self._has_tensor(value):
fixed_value = self._fix_tensor_device(
value, target_device
)
setattr(batch, attr_name, fixed_value)
except (AttributeError, TypeError):
continue
# DenoisingStage reads batch.scheduler. Native pipelines attach it in
# TimestepPreparationStage; ComfyUI already owns the timestep schedule.
get_or_create_request_scheduler(batch, self.scheduler)
original_latents_shape = None
if batch.latents is not None:
original_latents_shape = batch.latents.shape
# Call parent class's forward method
result = super().forward(batch, server_args)
if original_latents_shape is not None:
# Preserve the original shape before any potential packing/conversion
# (e.g., 4D spatial -> 3D sequence) to ensure proper unpadding later.
# Preserve the original shape before any packing/conversion
# (e.g., 4D spatial -> 3D sequence) so unpadding stays correct.
result.raw_latent_shape = original_latents_shape
return result
@@ -3,6 +3,7 @@
"""MiniMax H3-specific pipeline stages."""
from .stages.audio_encoding import MiniMaxH3AudioEncodingStage
from .stages.comfyui_step import MiniMaxH3ComfyUIStepStage
from .stages.decoding import MiniMaxH3DecodingStage
from .stages.denoising import MiniMaxH3DenoisingStage
from .stages.latent_preparation import MiniMaxH3LatentPreparationStage
@@ -12,6 +13,7 @@ from .stages.visual_encoding import MiniMaxH3VisualEncodingStage
__all__ = [
"MiniMaxH3AudioEncodingStage",
"MiniMaxH3ComfyUIStepStage",
"MiniMaxH3DecodingStage",
"MiniMaxH3DenoisingStage",
"MiniMaxH3LatentPreparationStage",
@@ -0,0 +1,799 @@
# SPDX-License-Identifier: Apache-2.0
"""ComfyUI per-step DiT stage for MiniMax H3.
ComfyUI owns the sampler loop. This stage runs one ``MiniMaxH3DiTModel.forward``
via ``MiniMaxH3DenoiseBranch.forward_kwargs`` and writes unpacked
``[video, audio]`` velocities onto ``batch.noise_pred``.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Any
import torch
_logger = logging.getLogger(__name__)
from sglang.multimodal_gen.configs.models.dits.minimax_h3 import (
MINIMAX_H3_ADALN_MODALITY_NUM,
MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT,
)
from sglang.multimodal_gen.runtime.models.dits.minimax_h3 import (
_FORWARD_SUPPORTED_KWARGS,
)
from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import (
bind_comfyui_session,
get_or_create_run_state,
get_run_state,
pop_run_state,
release_run_state,
session_id_from_req,
)
from sglang.multimodal_gen.runtime.pipelines_core.schedule_batch import Req
from sglang.multimodal_gen.runtime.pipelines_core.stages.base import PipelineStage
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.denoise_loop import (
MINIMAX_H3_AUDIO_REF_COND_TIMESTEP,
MINIMAX_H3_IMGVID_COND_TIMESTEP,
MiniMaxH3DenoiseBranch,
)
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.packed_sequence import (
minimax_h3_packed_sequence,
minimax_h3_packed_sequence_ref2va_blocks,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
DEFAULT_SIGMA_SHIFT_VIDEO = 12.0
DEFAULT_SIGMA_SHIFT_AUDIO = 3.0
_PATCH_SIZE = (1, 2, 2)
_SEG_TOKEN_TAG = {
"text": 1,
"video": 0,
"cond": 0,
"ref_img": 0,
"audio": 2,
"cond_audio": 2,
"ref_audio": 2,
}
def time_shift_sigma(sigma, from_shift, to_shift):
base = sigma / (from_shift + sigma * (1.0 - from_shift))
return to_shift * base / (1.0 + (to_shift - 1.0) * base)
def time_shift_slope(sigma, from_shift, to_shift):
"""d(sigma_to)/d(sigma_from) at the same base-grid point."""
base = sigma / (from_shift + sigma * (1.0 - from_shift))
return (to_shift * (1.0 + (from_shift - 1.0) * base) ** 2) / (
from_shift * (1.0 + (to_shift - 1.0) * base) ** 2
)
def patchify_video(latent, patch_size=_PATCH_SIZE):
b, c, t_full, h_full, w_full = latent.shape
pt, ph, pw = patch_size
t, h, w = t_full // pt, h_full // ph, w_full // pw
x = latent.reshape(b, c, t, pt, h, ph, w, pw)
x = torch.einsum("nctrhpwq->nthwcrpq", x)
return x.reshape(b * t * h * w, c * pt * ph * pw)
def unpatchify_video(rows, t, h, w, c=24, patch_size=_PATCH_SIZE):
pt, ph, pw = patch_size
x = rows.reshape(-1, t, h, w, c, pt, ph, pw)
x = torch.einsum("nthwcrpq->nctrhpwq", x)
return x.reshape(-1, c, t * pt, h * ph, w * pw)
def pack_audio(latent):
_b, _c, ch, t = latent.shape
return latent[0].permute(1, 2, 0).reshape(ch * t, latent.shape[1])
def unpack_audio(rows, ch=2):
t = rows.shape[0] // ch
return rows.reshape(ch, t, rows.shape[-1]).permute(2, 0, 1).unsqueeze(0)
def pad_to_patch_size(video, patch_size=_PATCH_SIZE):
pt, ph, pw = patch_size
t, h, w = video.shape[2], video.shape[3], video.shape[4]
pad_t = (pt - t % pt) % pt
pad_h = (ph - h % ph) % ph
pad_w = (pw - w % pw) % pw
if pad_t == pad_h == pad_w == 0:
return video
return torch.nn.functional.pad(video, (0, pad_w, 0, pad_h, 0, pad_t))
def _as_text_embeddings(context: torch.Tensor) -> torch.Tensor:
if context.ndim == 3:
return context[0]
return context
def _payload_has_keyframes(payload: dict[str, Any] | None) -> bool:
return bool((payload or {}).get("keyframes"))
def _payload_refs(payload: dict[str, Any] | None) -> list[dict[str, Any]]:
refs = (payload or {}).get("refs") or []
if not isinstance(refs, (list, tuple)):
raise TypeError(f"minimax_payload.refs must be a sequence, got {type(refs)!r}")
return list(refs)
def _apply_cond_noise(rows: torch.Tensor, aug: float, seed: int) -> torch.Tensor:
if aug >= 1.0:
return rows
gen = torch.Generator("cpu").manual_seed(int(seed))
noise = torch.randn(rows.shape, generator=gen, dtype=torch.float32)
return aug * rows + (1.0 - aug) * noise.to(rows.device)
def _ref_blocks_for_packed(refs: list[dict[str, Any]]) -> list[dict[str, Any]]:
blocks: list[dict[str, Any]] = []
for ref in refs:
block = {
key: value
for key, value in ref.items()
if key not in {"latent", "audio_latent"}
}
if "kind" not in block and "type" in block:
block["kind"] = block["type"]
blocks.append(block)
return blocks
def serialize_comfyui_layout(layout: Any) -> dict[str, Any] | None:
"""Turn a ComfyUI ``PackedLayout`` into pickle/IPC-safe tensors."""
required = (
"seq_len",
"signature",
"segments",
"position_ids",
"img_pos",
"img_update",
"audio_pos",
"audio_update",
)
if layout is None or any(not hasattr(layout, name) for name in required):
return None
return {
"seq_len": int(layout.seq_len),
"signature": tuple(int(value) for value in layout.signature),
"segments": [
(int(start), int(stop), str(kind)) for start, stop, kind in layout.segments
],
"position_ids": layout.position_ids.detach().cpu().to(torch.float64),
"img_pos": layout.img_pos.detach().cpu().to(torch.long),
"img_update": layout.img_update.detach().cpu().to(torch.bool),
"audio_pos": layout.audio_pos.detach().cpu().to(torch.long),
"audio_update": layout.audio_update.detach().cpu().to(torch.bool),
}
def _layout_signature_matches(
layout: dict[str, Any],
video_x: torch.Tensor,
audio_x: torch.Tensor,
text: torch.Tensor,
) -> bool:
"""ComfyUI ``PackedLayout.signature`` is ``(text, T, H, W, audio_t)``.
A cached layout from an earlier sampler pass (e.g. pre-upscale 14×8
tokens) must not be applied to a different spatial grid.
"""
sig = layout.get("signature")
if sig is None:
return False
values = tuple(int(v) for v in sig[:5])
if len(values) < 5:
return False
text_len, latent_t, latent_h, latent_w, audio_t = values
return (
text_len == int(text.shape[0])
and latent_t == int(video_x.shape[2])
and latent_h == int(video_x.shape[3])
and latent_w == int(video_x.shape[4])
and audio_t == int(audio_x.shape[-1])
)
def comfyui_layout_to_packed(layout: dict[str, Any]) -> dict[str, Any]:
"""Rebuild the SGLD packed dict from a serialized ComfyUI layout.
Only pads ``seq_len`` to 64. Row order, RoPE ids, and update masks stay
exactly as ComfyUI computed them.
"""
used = int(layout["seq_len"])
seq_len = (
(used + MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT - 1)
// MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT
* MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT
)
position_ids = layout["position_ids"].to(dtype=torch.float64)
grid = torch.zeros(seq_len, 3, dtype=torch.float64)
grid[:used] = position_ids[:used]
token_tags = torch.full((seq_len,), -1, dtype=torch.long)
text_pos = torch.empty(0, dtype=torch.long)
for start, stop, kind in layout["segments"]:
token_tags[start:stop] = _SEG_TOKEN_TAG.get(kind, -1)
if kind == "text":
text_pos = torch.arange(start, stop, dtype=torch.long)
return {
"seq_len": seq_len,
"img_pos": layout["img_pos"].view(-1).to(torch.long),
"audio_pos": layout["audio_pos"].view(-1).to(torch.long),
"text_pos": text_pos,
"update_mask": layout["img_update"].view(-1).to(torch.bool),
"audio_update_mask": layout["audio_update"].view(-1).to(torch.bool),
"img_position_ids": grid,
"token_tags": token_tags,
"cu_seqlens": torch.tensor([0, used, seq_len], dtype=torch.int32),
}
def _latent_items(payload: dict[str, Any], key: str) -> list[Any]:
items = payload.get(key) or []
if not isinstance(items, (list, tuple)):
return [items]
return [item for item in items if item is not None]
def _assemble_media_rows(
video: torch.Tensor,
audio: torch.Tensor,
payload: dict[str, Any],
) -> tuple[torch.Tensor, torch.Tensor]:
video_target = patchify_video(video, _PATCH_SIZE)
audio_target = pack_audio(audio)
vis_aug = float(
payload.get("visual_cond_noise_aug", MINIMAX_H3_IMGVID_COND_TIMESTEP)
)
aud_aug = float(
payload.get("audio_cond_noise_aug", MINIMAX_H3_AUDIO_REF_COND_TIMESTEP)
)
seed = int(payload.get("seed", 0))
cond_video_src = list(_latent_items(payload, "cond_video_latents"))
cond_audio_src = list(_latent_items(payload, "cond_audio_latents"))
if not cond_video_src and not cond_audio_src:
for block in list(payload.get("keyframes") or []) + _payload_refs(payload):
if block.get("latent") is not None:
cond_video_src.append(block["latent"])
if block.get("audio_latent") is not None:
cond_audio_src.append(block["audio_latent"])
cond_video: list[torch.Tensor] = []
for latent in cond_video_src:
rows = patchify_video(
pad_to_patch_size(
latent.to(device=video.device, dtype=torch.float32),
_PATCH_SIZE,
),
_PATCH_SIZE,
)
cond_video.append(_apply_cond_noise(rows, vis_aug, seed))
cond_audio: list[torch.Tensor] = []
for latent in cond_audio_src:
rows = pack_audio(latent.to(device=audio.device, dtype=torch.float32))
cond_audio.append(_apply_cond_noise(rows, aud_aug, seed + 1))
video_rows = (
torch.cat(cond_video + [video_target], dim=0) if cond_video else video_target
)
audio_rows = (
torch.cat(cond_audio + [audio_target], dim=0) if cond_audio else audio_target
)
return video_rows, audio_rows
def _token_tags_for_payload(
packed: dict[str, torch.Tensor],
payload: dict[str, Any],
) -> torch.Tensor:
tags = packed["token_tags"]
text_tags = payload.get("text_token_tags")
if text_tags is None:
return tags
text_pos = packed["text_pos"].view(-1)
text_tags = text_tags.reshape(-1).to(dtype=torch.long)
if int(text_tags.numel()) != int(text_pos.numel()):
raise ValueError(
"text_token_tags length "
f"{int(text_tags.numel())} != packed text_len {int(text_pos.numel())}"
)
tags = tags.clone()
tags[text_pos] = text_tags
return tags
def _sigma_list(sample_sigmas: Any) -> list[float] | None:
n = _sample_sigmas_len(sample_sigmas)
if n is None:
return None
if torch.is_tensor(sample_sigmas):
return [float(v) for v in sample_sigmas.reshape(-1).tolist()]
return [float(v) for v in sample_sigmas]
def _as_spatial_mask(mask: Any) -> torch.Tensor | None:
if mask is None:
return None
values = mask
if not torch.is_tensor(values):
values = torch.as_tensor(values)
if int(values.numel()) == 0:
return None
if values.ndim == 5:
values = values[0, 0]
elif values.ndim == 4:
values = values[0]
elif values.ndim == 2:
values = values.unsqueeze(0)
return values.to(dtype=torch.float32)
def mask_row_values(mask: Any, latent_t: int, lat_h: int, lat_w: int):
"""Match ComfyUI ``mask_row_values``: [T,H,W] → one float per 2×2 patch row."""
values = _as_spatial_mask(mask)
if values is None:
return None
if values.ndim != 3:
raise ValueError(
f"denoise_mask must be [T,H,W] after squeeze, got {tuple(values.shape)}"
)
if values.shape[0] == 1 and latent_t > 1:
values = values.expand(latent_t, *values.shape[1:])
elif values.shape[0] != latent_t:
raise ValueError(
f"denoise_mask T={int(values.shape[0])} != padded latent T={latent_t}"
)
pad_w = lat_w - int(values.shape[-1])
pad_h = lat_h - int(values.shape[-2])
if pad_w > 0 or pad_h > 0:
values = torch.nn.functional.pad(
values, (0, max(pad_w, 0), 0, max(pad_h, 0)), mode="replicate"
)
values = values[:latent_t, :lat_h, :lat_w]
values = values.reshape(latent_t, lat_h // 2, 2, lat_w // 2, 2).amax(dim=(2, 4))
rows = values.reshape(-1)
if bool((rows >= 1.0 - 1e-3).all()):
return None
return rows
def _audio_mask_values(mask: Any):
if mask is None:
return None
values = mask.reshape(-1).to(dtype=torch.float32)
if int(values.numel()) == 0:
return None
if bool((values >= 1.0 - 1e-3).all()):
return None
return values
def _overlay_per_row_timesteps(
fk: dict[str, Any],
branch: MiniMaxH3DenoiseBranch,
video_rows_t: torch.Tensor | None,
audio_rows_t: torch.Tensor | None,
) -> dict[str, Any]:
"""Apply ComfyUI per-row denoise_mask only on target video/audio rows."""
if video_rows_t is None and audio_rows_t is None:
return fk
unique = fk["unique_timesteps"].to(torch.float32)
inverse = fk["inverse_indices"].clone()
device = inverse.device
extras = [unique.reshape(-1).detach().cpu()]
if video_rows_t is not None:
extras.append(video_rows_t.reshape(-1).to(torch.float32).cpu())
if audio_rows_t is not None:
extras.append(audio_rows_t.reshape(-1).to(torch.float32).cpu())
new_unique = torch.unique(torch.cat(extras), sorted=True)
inverse = torch.searchsorted(new_unique, unique.cpu()).to(device)[inverse]
if video_rows_t is not None:
mapped = torch.searchsorted(
new_unique, video_rows_t.reshape(-1).to(torch.float32).cpu()
)
tgt = branch.img_target_seq_idx
if int(mapped.numel()) != int(tgt.numel()):
raise ValueError(
"denoise_mask rows "
f"{int(mapped.numel())} != video target rows {int(tgt.numel())}"
)
inverse[tgt] = mapped.to(device)
if audio_rows_t is not None:
mapped = torch.searchsorted(
new_unique, audio_rows_t.reshape(-1).to(torch.float32).cpu()
)
tgt = branch.audio_target_seq_idx
if int(mapped.numel()) != int(tgt.numel()):
raise ValueError(
"audio_denoise_mask rows "
f"{int(mapped.numel())} != audio target rows {int(tgt.numel())}"
)
inverse[tgt] = mapped.to(device)
fk["unique_timesteps"] = new_unique.to(device=device, dtype=unique.dtype)
fk["inverse_indices"] = inverse
fk["block_combined_indices"] = (
branch.block_token_tags
+ inverse[branch.local_row_slice] * MINIMAX_H3_ADALN_MODALITY_NUM
)
return fk
def comfyui_payload_to_branch_inputs(
video_x: torch.Tensor,
audio_x: torch.Tensor,
context: torch.Tensor,
minimax_payload: dict[str, Any] | None,
sample_sigmas: Any,
timestep: torch.Tensor,
transformer_options: dict[str, Any] | None = None,
layout: dict[str, Any] | None = None,
denoise_mask: Any = None,
audio_denoise_mask: Any = None,
) -> dict[str, Any]:
"""Translate one ComfyUI H3 step into Branch + ``forward_kwargs`` inputs.
``context`` is raw Qwen states (last dim 5120). Token refinement is
precomputed once on the worker after the Branch is created.
"""
payload = minimax_payload or {}
orig_shape = tuple(video_x.shape)
video = pad_to_patch_size(video_x.to(torch.float32), _PATCH_SIZE)
audio = audio_x.to(torch.float32)
text = _as_text_embeddings(context)
if layout is not None and not _layout_signature_matches(
layout, video_x, audio, text
):
_logger.warning(
"Ignoring stale ComfyUI PackedLayout signature %s for video %s "
"audio %s text_len %s",
layout.get("signature"),
tuple(video_x.shape),
tuple(audio.shape),
int(text.shape[0]),
)
layout = None
if layout is None and _payload_has_keyframes(payload):
raise NotImplementedError(
"MiniMaxH3 ComfyUI step needs a serialized PackedLayout for fl2va keyframes"
)
text_len = int(text.shape[0])
latent_t, latent_h, latent_w = (
int(video.shape[2]),
int(video.shape[3]),
int(video.shape[4]),
)
audio_t = int(audio.shape[-1])
refs = _payload_refs(payload)
if layout is not None:
packed = comfyui_layout_to_packed(layout)
elif refs:
packed = minimax_h3_packed_sequence_ref2va_blocks(
text_len=text_len,
latent_t=latent_t,
latent_h=latent_h,
latent_w=latent_w,
audio_t=audio_t,
ref_blocks=_ref_blocks_for_packed(refs),
)
else:
packed = minimax_h3_packed_sequence(
text_len=text_len,
latent_t=latent_t,
latent_h=latent_h,
latent_w=latent_w,
audio_t=audio_t,
include_keyframe_cond=False,
)
used = int(packed["cu_seqlens"][1])
video_rows, audio_rows = _assemble_media_rows(video, audio, payload)
opts = transformer_options or {}
shift_v = float(opts.get("minimax_h3_sigma_shift_video", DEFAULT_SIGMA_SHIFT_VIDEO))
shift_a = float(opts.get("minimax_h3_sigma_shift_audio", DEFAULT_SIGMA_SHIFT_AUDIO))
sigma_v = (timestep.reshape(-1)[0] / 1000.0).to(torch.float32).clamp(min=1e-6)
sigma_a = time_shift_sigma(sigma_v, shift_v, shift_a)
t_video = float(1.0 - sigma_v)
t_audio = float(1.0 - sigma_a)
t_pin_v = max(t_video, MINIMAX_H3_IMGVID_COND_TIMESTEP)
t_pin_a = max(t_audio, MINIMAX_H3_AUDIO_REF_COND_TIMESTEP)
# ComfyUI only remaps the video/audio *target* segments. Text / pad keep t_video.
video_rows_t = mask_row_values(denoise_mask, latent_t, latent_h, latent_w)
if video_rows_t is not None:
video_rows_t = (1.0 - video_rows_t * float(sigma_v)).clamp(max=t_pin_v)
audio_rows_t = _audio_mask_values(audio_denoise_mask)
if audio_rows_t is not None:
audio_rows_t = (1.0 - audio_rows_t * float(sigma_a)).clamp(max=t_pin_a)
return {
"packed": packed,
"text_embeddings": text,
"token_tags": _token_tags_for_payload(packed, payload),
"video_rows": video_rows,
"audio_rows": audio_rows,
"t_video": t_video,
"t_audio": t_audio,
"video_rows_t": video_rows_t,
"audio_rows_t": audio_rows_t,
"used": used,
"orig_video_shape": orig_shape,
"padded_video_shape": tuple(video.shape),
"sample_sigmas": sample_sigmas,
"sigma_v": float(sigma_v),
"shift_v": shift_v,
"shift_a": shift_a,
"imgvid_cond_noise_aug": float(
payload.get("visual_cond_noise_aug", MINIMAX_H3_IMGVID_COND_TIMESTEP)
),
"audio_ref_cond_noise_aug": float(
payload.get("audio_cond_noise_aug", MINIMAX_H3_AUDIO_REF_COND_TIMESTEP)
),
}
def build_step_forward_kwargs(
inputs: dict[str, Any],
*,
branch: MiniMaxH3DenoiseBranch | None = None,
device: torch.device | None = None,
) -> tuple[dict[str, Any], MiniMaxH3DenoiseBranch]:
"""Assemble DiT ``forward`` kwargs from ``comfyui_payload_to_branch_inputs``."""
if device is None:
device = inputs["video_rows"].device
if branch is None:
branch = MiniMaxH3DenoiseBranch(
packed=inputs["packed"],
text_embeddings=inputs["text_embeddings"],
token_tags=inputs["token_tags"],
device=device,
)
plan = branch.prepare_timestep_plan(
video_timesteps=[inputs["t_video"]],
audio_timesteps=[inputs["t_audio"]],
imgvid_cond_noise_aug=inputs["imgvid_cond_noise_aug"],
audio_ref_cond_noise_aug=inputs["audio_ref_cond_noise_aug"],
)
fk = branch.forward_kwargs(
video_rows=inputs["video_rows"].to(device=device, dtype=torch.float32),
audio_rows=inputs["audio_rows"].to(device=device, dtype=torch.float32),
step_timesteps=plan[0],
)
fk = _overlay_per_row_timesteps(
fk, branch, inputs.get("video_rows_t"), inputs.get("audio_rows_t")
)
extra = set(fk) - _FORWARD_SUPPORTED_KWARGS
if extra:
raise TypeError(
f"MiniMaxH3 ComfyUI step produced unsupported forward kwargs: {sorted(extra)}"
)
return fk, branch
def pack_comfy_output(
v_video: torch.Tensor,
v_audio: torch.Tensor,
state: MiniMaxH3ComfyUIRunState,
) -> list[torch.Tensor]:
_b, _c, orig_t, orig_h, orig_w = state.orig_video_shape
_pb, _pc, padded_t, padded_h, padded_w = state.padded_video_shape
video = unpatchify_video(
v_video,
padded_t,
padded_h // 2,
padded_w // 2,
c=int(state.orig_video_shape[1]),
patch_size=_PATCH_SIZE,
)
video = video[:, :, :orig_t, :orig_h, :orig_w]
# DiT returns target video rows, but audio_pos includes reference rows.
audio = unpack_audio(v_audio[state.branch.audio_target_slice])
return [video, audio]
def _sample_sigmas_len(sample_sigmas: Any) -> int | None:
if sample_sigmas is None:
return None
if torch.is_tensor(sample_sigmas):
return int(sample_sigmas.reshape(-1).numel())
if isinstance(sample_sigmas, (list, tuple)):
return len(sample_sigmas)
return None
@dataclass
class MiniMaxH3ComfyUIRunState:
branch: MiniMaxH3DenoiseBranch
used: int
orig_video_shape: tuple[int, ...]
padded_video_shape: tuple[int, ...]
sample_sigmas: Any = None
cache_mounted: bool = False
refined: bool = False
steps_done: int = 0
signature: tuple[Any, ...] | None = None
class MiniMaxH3ComfyUIStepStage(PipelineStage):
def __init__(self, transformer) -> None:
super().__init__()
self.transformer = transformer
self._cache_stage = None
def verify_input(self, batch: Req, server_args: ServerArgs):
bind_comfyui_session(batch)
return super().verify_input(batch, server_args)
def _cache_dit_stage(self):
if self._cache_stage is None:
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.denoising import (
MiniMaxH3DenoisingStage,
)
self._cache_stage = MiniMaxH3DenoisingStage(transformer=self.transformer)
return self._cache_stage
def _maybe_mount_cache_dit(
self, batch: Req, state: MiniMaxH3ComfyUIRunState
) -> None:
if state.cache_mounted:
return
n_sigmas = _sample_sigmas_len(state.sample_sigmas)
if n_sigmas is None or n_sigmas < 2:
return
self._cache_dit_stage()._maybe_enable_cache_dit(n_sigmas - 1, batch)
state.cache_mounted = True
def _release_run_state(self, state: MiniMaxH3ComfyUIRunState | None) -> None:
if state is None:
return
if state.cache_mounted and self._cache_stage is not None:
self._cache_stage._unmount_cache_dit()
state.cache_mounted = False
state.branch = None
def _read_step_tensors(
self, batch: Req
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
extra = getattr(batch, "extra", None) or {}
video = batch.latents
audio = batch.audio_latents
if audio is None:
audio = extra.get("audio_latents")
context = extra.get("h3_context")
if context is None:
embeds = batch.prompt_embeds
if isinstance(embeds, list) and embeds:
context = embeds[0]
else:
context = embeds
if video is None or audio is None or context is None:
raise ValueError(
"MiniMaxH3 ComfyUI step requires latents, audio_latents, and prompt_embeds"
)
return video, audio, context
def _bind_run_state(
self,
batch: Req,
inputs: dict[str, Any],
sample_sigmas: Any,
device: torch.device,
) -> MiniMaxH3ComfyUIRunState:
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.denoising import (
_precompute_refined_prompt_embeds,
)
signature = (
inputs["orig_video_shape"],
inputs["used"],
int(inputs["text_embeddings"].shape[0]),
)
existing = get_run_state(batch)
sigmas = _sigma_list(sample_sigmas)
is_first_sigma = False
if sigmas:
is_first_sigma = abs(float(inputs["sigma_v"]) - sigmas[0]) < 1e-4
if existing is not None and (
existing.signature != signature
or (is_first_sigma and existing.steps_done > 0)
):
self._release_run_state(existing)
pop_run_state(batch)
existing = None
def _factory() -> MiniMaxH3ComfyUIRunState:
branch = MiniMaxH3DenoiseBranch(
packed=inputs["packed"],
text_embeddings=inputs["text_embeddings"],
token_tags=inputs["token_tags"],
device=device,
)
return MiniMaxH3ComfyUIRunState(
branch=branch,
used=inputs["used"],
orig_video_shape=inputs["orig_video_shape"],
padded_video_shape=inputs["padded_video_shape"],
sample_sigmas=sample_sigmas,
signature=signature,
)
state = (
existing
if existing is not None
else get_or_create_run_state(batch, _factory)
)
if not state.refined:
_precompute_refined_prompt_embeds(
self.transformer, state.branch, device=device
)
state.refined = True
return state
def forward(self, batch: Req, server_args: ServerArgs) -> Req:
bind_comfyui_session(batch)
extra = getattr(batch, "extra", None) or {}
video_x, audio_x, context = self._read_step_tensors(batch)
payload = extra.get("h3_payload") or {}
transformer_options = extra.get("h3_transformer_options") or {}
sample_sigmas = extra.get("h3_sample_sigmas")
if sample_sigmas is None:
sample_sigmas = batch.sigmas
timestep = batch.timesteps
if timestep is None:
timestep = batch.timestep
if timestep is None:
raise ValueError("MiniMaxH3 ComfyUI step requires timesteps")
inputs = comfyui_payload_to_branch_inputs(
video_x,
audio_x,
context,
payload,
sample_sigmas,
timestep,
transformer_options=transformer_options,
layout=extra.get("h3_layout"),
denoise_mask=extra.get("h3_denoise_mask"),
audio_denoise_mask=extra.get("h3_audio_denoise_mask"),
)
state = self._bind_run_state(batch, inputs, sample_sigmas, video_x.device)
self._maybe_mount_cache_dit(batch, state)
fk, _branch = build_step_forward_kwargs(
inputs, branch=state.branch, device=video_x.device
)
v_video, v_audio = self.transformer(**fk)
batch.noise_pred = pack_comfy_output(v_video, v_audio, state)
state.steps_done += 1
n_sigmas = _sample_sigmas_len(sample_sigmas)
if n_sigmas is not None and n_sigmas >= 2 and state.steps_done >= n_sigmas - 1:
self._release_run_state(state)
release_run_state(session_id_from_req(batch))
return batch
__all__ = [
"DEFAULT_SIGMA_SHIFT_AUDIO",
"DEFAULT_SIGMA_SHIFT_VIDEO",
"MiniMaxH3ComfyUIRunState",
"MiniMaxH3ComfyUIStepStage",
"build_step_forward_kwargs",
"comfyui_layout_to_packed",
"comfyui_payload_to_branch_inputs",
"mask_row_values",
"pack_audio",
"pack_comfy_output",
"patchify_video",
"serialize_comfyui_layout",
"time_shift_sigma",
"time_shift_slope",
"unpack_audio",
"unpatchify_video",
]
@@ -7,6 +7,11 @@ from typing import Any, Optional
import zmq
import zmq.asyncio
from sglang.multimodal_gen.runtime.distributed.ipc_cuda import (
materialize_cuda_refs,
release_retained_producer_tensors,
spill_cuda_tensors,
)
from sglang.multimodal_gen.runtime.entrypoints.control_requests import (
ListLorasReq,
MergeLoraWeightsReq,
@@ -22,7 +27,10 @@ from sglang.multimodal_gen.runtime.entrypoints.post_training.io_struct import (
UpdateWeightFromTensorCheckerReqInput,
UpdateWeightFromTensorReqInput,
)
from sglang.multimodal_gen.runtime.ipc_array import materialize_file_refs
from sglang.multimodal_gen.runtime.ipc_array import (
is_local_endpoint,
materialize_file_refs,
)
from sglang.multimodal_gen.runtime.pipelines_core import Req
from sglang.multimodal_gen.runtime.pipelines_core.schedule_batch import OutputBatch
from sglang.multimodal_gen.runtime.server_args import (
@@ -174,9 +182,10 @@ class SchedulerClient:
effective_timeout = _resolve_timeout_ms(self.server_args, timeout_ms)
_configure_recv_timeout(socket, effective_timeout)
socket.connect(endpoint)
socket.send_pyobj(batch)
socket.send_pyobj(_prepare_local_payload(endpoint, batch))
output_batch = socket.recv_pyobj()
_materialize_output_batch_file_refs(output_batch)
_materialize_local_cuda_refs(endpoint, output_batch)
return output_batch
except zmq.error.Again:
logger.error("Timeout waiting for response from %s.", endpoint)
@@ -286,10 +295,11 @@ class AsyncSchedulerClient:
effective_timeout = _resolve_timeout_ms(self.server_args, timeout_ms)
_configure_recv_timeout(socket, effective_timeout)
socket.connect(endpoint)
await socket.send(pickle.dumps(batch))
await socket.send(pickle.dumps(_prepare_local_payload(endpoint, batch)))
payload = await socket.recv()
output_batch = pickle.loads(payload)
_materialize_output_batch_file_refs(output_batch)
_materialize_local_cuda_refs(endpoint, output_batch)
return output_batch
except zmq.error.Again:
logger.error("Timeout waiting for response from %s.", endpoint)
@@ -331,6 +341,19 @@ async_scheduler_client = AsyncSchedulerClient()
sync_scheduler_client = SchedulerClient()
def _prepare_local_payload(endpoint: str, batch: Any) -> Any:
if is_local_endpoint(endpoint):
return spill_cuda_tensors(batch)
return batch
def _materialize_local_cuda_refs(endpoint: str, output_batch: Any) -> None:
if is_local_endpoint(endpoint):
materialize_cuda_refs(output_batch)
# Request-side IPC retains are no longer needed after the round-trip.
release_retained_producer_tensors()
def _materialize_output_batch_file_refs(output_batch: Any) -> None:
if not isinstance(output_batch, OutputBatch):
return
@@ -0,0 +1,58 @@
# SPDX-License-Identifier: Apache-2.0
"""Pack / unpack contract for ComfyUI model adapters."""
import torch
from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.adapter import (
get_adapter_class,
registered_model_types,
)
from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.flux import FluxAdapter
from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.zimage import (
ZImageAdapter,
)
def test_registered_comfyui_model_types() -> None:
types = registered_model_types()
assert "flux" in types
assert "lumina2" in types
assert get_adapter_class("lumina2") is ZImageAdapter
assert get_adapter_class("flux") is FluxAdapter
assert get_adapter_class("lumina2").pipeline_class_name == "ZImagePipeline"
def test_zimage_pack_sets_seq_lens_and_time_dim() -> None:
adapter = ZImageAdapter()
x = torch.ones(1, 16, 90, 160)
timestep = torch.tensor([1.0])
context = torch.ones(1, 19, 2560)
packed = adapter.pack(x, timestep, context)
assert packed.latents.shape == (1, 16, 1, 90, 160)
assert packed.prompt_embeds[0].shape == (19, 2560)
assert packed.prompt_seq_lens == [[19]]
assert packed.height == 720
assert packed.width == 1280
assert torch.equal(packed.timesteps, timestep * 1000.0)
pred = torch.ones(1, 16, 1, 90, 160)
out = adapter.unpack(pred, packed, x)
assert out.shape == x.shape
def test_flux_pack_and_unpack_roundtrip() -> None:
adapter = FluxAdapter()
x = torch.arange(1 * 16 * 8 * 8, dtype=torch.float32).reshape(1, 16, 8, 8)
timestep = torch.tensor([0.5])
context = torch.ones(1, 8, 4096)
y = torch.ones(1, 768)
packed = adapter.pack(x, timestep, context, y=y, guidance=torch.tensor([1.0]))
assert packed.latents.ndim == 3
assert packed.pooled_embeds[0] is y
assert packed.guidance_scale == 1.0
out = adapter.unpack(packed.latents, packed, x)
assert out.shape == x.shape
assert torch.equal(out, x)
default = adapter.pack(x, timestep, context, y=y)
assert default.guidance_scale == 3.5
@@ -0,0 +1,85 @@
# SPDX-License-Identifier: Apache-2.0
"""Process-wide SGLD worker ownership for ComfyUI loaders."""
from types import SimpleNamespace
from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.core.generator import (
SGLDiffusionGenerator,
)
def test_shared_is_process_singleton() -> None:
SGLDiffusionGenerator.reset_shared()
assert SGLDiffusionGenerator.shared() is SGLDiffusionGenerator.shared()
SGLDiffusionGenerator.reset_shared()
def test_reuse_requires_live_worker() -> None:
runtime = SGLDiffusionGenerator()
options = {"model_path": "z.safetensors"}
runtime.last_options = options
runtime.generator = object()
runtime._patcher = object()
runtime.executor = object()
runtime._is_live = lambda: False
assert runtime._can_reuse(options) is False
runtime._is_live = lambda: True
assert runtime._can_reuse(options) is True
assert runtime._can_reuse({"model_path": "h3.safetensors"}) is False
def test_ensure_rebuilds_when_another_model_owns_the_worker() -> None:
runtime = SGLDiffusionGenerator()
stale = object()
fresh = object()
loads = []
executor = SimpleNamespace(
generator=stale,
_sgld_reload={
"model_path": "z.safetensors",
"model_options": {},
"sgld_options": {},
},
_lora_input=None,
)
def fake_load(**kwargs):
loads.append(kwargs)
runtime.generator = fresh
return "patcher"
runtime.load_model = fake_load
runtime.ensure_executor(executor)
assert loads == [executor._sgld_reload]
assert executor.generator is fresh
def test_ensure_is_noop_when_executor_still_owns_live_worker() -> None:
runtime = SGLDiffusionGenerator()
gen = object()
runtime.generator = gen
runtime._is_live = lambda: True
executor = SimpleNamespace(generator=gen, _sgld_reload={"model_path": "z"})
runtime.load_model = lambda **kwargs: (_ for _ in ()).throw(
AssertionError("should not reload")
)
runtime.ensure_executor(executor)
assert executor.generator is gen
def test_kill_generator_only_touches_owned_workers() -> None:
runtime = SGLDiffusionGenerator()
owned = SimpleNamespace(alive=True, terminated=False, killed=False, pid=9)
def terminate():
owned.terminated = True
owned.alive = False
owned.is_alive = lambda: owned.alive
owned.terminate = terminate
owned.join = lambda timeout=None: None
owned.kill = lambda: setattr(owned, "killed", True)
runtime.generator = SimpleNamespace(local_scheduler_process=[owned])
runtime.kill_generator()
assert owned.terminated is True
assert owned.killed is False
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,205 @@
# SPDX-License-Identifier: Apache-2.0
"""Tests for the comfyui_mode profile of native pipelines.
The existing ComfyUI tests under apps/ComfyUI_SGLDiffusion/test need a real
multi-GB checkpoint and two GPUs, so nothing here is covered in CI. These build
a tiny ComfyUI-format checkpoint instead, which is enough to exercise module
trimming, single-file loading, weight conversion, and stage construction.
"""
import os
import tempfile
import pytest
import torch
from safetensors.torch import save_file
from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig
from sglang.multimodal_gen.registry import get_pipeline_class
from sglang.multimodal_gen.runtime.distributed.parallel_state import (
maybe_init_distributed_environment_and_model_parallel,
model_parallel_is_initialized,
)
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import (
get_comfyui_checkpoint_spec,
)
from sglang.multimodal_gen.runtime.server_args import ServerArgs
from sglang.multimodal_gen.runtime.server_args.server_args import set_global_server_args
from sglang.multimodal_gen.test.single_test_file.component_accuracy.utils import (
ensure_distributed_env_defaults,
)
HEADS, HEAD_DIM, LAYERS, SINGLE_LAYERS = 4, 16, 2, 2
def _ensure_single_process_parallel_runtime() -> None:
if model_parallel_is_initialized():
return
ensure_distributed_env_defaults()
maybe_init_distributed_environment_and_model_parallel(tp_size=1, sp_size=1)
def _shrink(arch) -> None:
arch.num_attention_heads = HEADS
arch.attention_head_dim = HEAD_DIM
arch.num_layers = LAYERS
arch.num_single_layers = SINGLE_LAYERS
arch.guidance_embeds = True
def _write_comfyui_flux_checkpoint(path: str, config: FluxConfig) -> dict:
"""Write a checkpoint using ComfyUI's parameter names and fused layouts."""
arch = config.arch_config
hidden = arch.num_attention_heads * arch.attention_head_dim
mlp_hidden = int(hidden * getattr(arch, "mlp_ratio", 4.0))
qkv = 3 * hidden
out_channels = arch.out_channels or arch.in_channels
patch = getattr(arch, "patch_size", 1)
sd: dict[str, torch.Tensor] = {}
def put(name: str, *shape: int) -> None:
sd[name] = torch.randn(*shape, dtype=torch.bfloat16)
for b in range(arch.num_layers):
for attn in ("img_attn", "txt_attn"):
put(f"double_blocks.{b}.{attn}.qkv.weight", qkv, hidden)
put(f"double_blocks.{b}.{attn}.qkv.bias", qkv)
put(f"double_blocks.{b}.{attn}.proj.weight", hidden, hidden)
put(f"double_blocks.{b}.{attn}.proj.bias", hidden)
put(
f"double_blocks.{b}.{attn}.norm.query_norm.scale",
arch.attention_head_dim,
)
put(
f"double_blocks.{b}.{attn}.norm.key_norm.scale", arch.attention_head_dim
)
for mlp in ("img_mlp", "txt_mlp"):
put(f"double_blocks.{b}.{mlp}.0.weight", mlp_hidden, hidden)
put(f"double_blocks.{b}.{mlp}.0.bias", mlp_hidden)
put(f"double_blocks.{b}.{mlp}.2.weight", hidden, mlp_hidden)
put(f"double_blocks.{b}.{mlp}.2.bias", hidden)
for mod in ("img_mod", "txt_mod"):
put(f"double_blocks.{b}.{mod}.lin.weight", 6 * hidden, hidden)
put(f"double_blocks.{b}.{mod}.lin.bias", 6 * hidden)
for b in range(arch.num_single_layers):
put(f"single_blocks.{b}.linear1.weight", qkv + mlp_hidden, hidden)
put(f"single_blocks.{b}.linear1.bias", qkv + mlp_hidden)
put(f"single_blocks.{b}.linear2.weight", hidden, hidden + mlp_hidden)
put(f"single_blocks.{b}.linear2.bias", hidden)
put(f"single_blocks.{b}.norm.query_norm.scale", arch.attention_head_dim)
put(f"single_blocks.{b}.norm.key_norm.scale", arch.attention_head_dim)
put(f"single_blocks.{b}.modulation.lin.weight", 3 * hidden, hidden)
put(f"single_blocks.{b}.modulation.lin.bias", 3 * hidden)
for stem, in_dim in (
("time_in", 256),
("vector_in", arch.pooled_projection_dim),
("guidance_in", 256),
):
put(f"{stem}.in_layer.weight", hidden, in_dim)
put(f"{stem}.in_layer.bias", hidden)
put(f"{stem}.out_layer.weight", hidden, hidden)
put(f"{stem}.out_layer.bias", hidden)
put("txt_in.weight", hidden, arch.joint_attention_dim)
put("txt_in.bias", hidden)
put("img_in.weight", hidden, arch.in_channels)
put("img_in.bias", hidden)
put("final_layer.linear.weight", patch * patch * out_channels, hidden)
put("final_layer.linear.bias", patch * patch * out_channels)
put("final_layer.adaLN_modulation.1.weight", 2 * hidden, hidden)
put("final_layer.adaLN_modulation.1.bias", 2 * hidden)
save_file(sd, path)
return sd
@pytest.fixture(scope="module")
def comfyui_flux_pipeline():
if not torch.cuda.is_available():
pytest.skip("requires CUDA")
tmpdir = tempfile.mkdtemp(prefix="comfyui_flux_")
checkpoint = os.path.join(tmpdir, "flux_comfyui.safetensors")
# The file has to exist before ServerArgs resolves the single-file path.
seed_config = FluxConfig()
_shrink(seed_config.arch_config)
state_dict = _write_comfyui_flux_checkpoint(checkpoint, seed_config)
server_args = ServerArgs.from_kwargs(
model_path=checkpoint,
pipeline_class_name="FluxPipeline",
comfyui_mode=True,
num_gpus=1,
)
_shrink(server_args.pipeline_config.dit_config.arch_config)
set_global_server_args(server_args)
_ensure_single_process_parallel_runtime()
pipeline = get_pipeline_class("FluxPipeline")(
model_path=checkpoint, server_args=server_args
)
return pipeline, state_dict
def test_specs_cover_every_comfyui_supported_pipeline():
for pipeline_name in (
"FluxPipeline",
"ZImagePipeline",
"QwenImagePipeline",
"QwenImageEditPlusPipeline",
):
spec = get_comfyui_checkpoint_spec(pipeline_name)
assert spec is not None, f"{pipeline_name} has no ComfyUI checkpoint spec"
assert get_pipeline_class(pipeline_name) is not None
assert get_pipeline_class(pipeline_name).pipeline_config_cls is not None
def test_comfyui_mode_trims_pipeline_to_a_dit_forward_service(comfyui_flux_pipeline):
pipeline, _ = comfyui_flux_pipeline
assert pipeline.required_config_modules == ["transformer", "scheduler"]
assert (
type(pipeline.get_module("scheduler")).__name__ == "ComfyUIPassThroughScheduler"
)
assert list(pipeline._stage_name_mapping) == [
"ComfyUILatentPreparationStage",
"DenoisingStage",
]
def test_comfyui_checkpoint_fully_populates_the_transformer(comfyui_flux_pipeline):
pipeline, _ = comfyui_flux_pipeline
transformer = pipeline.get_module("transformer")
unloaded = [name for name, p in transformer.named_parameters() if p.is_meta]
assert not unloaded, f"parameters never received checkpoint weights: {unloaded[:5]}"
def test_fused_qkv_is_split_into_separate_projections(comfyui_flux_pipeline):
pipeline, state_dict = comfyui_flux_pipeline
params = dict(pipeline.get_module("transformer").named_parameters())
device = next(iter(params.values())).device
hidden = HEADS * HEAD_DIM
fused = state_dict["double_blocks.0.img_attn.qkv.weight"].to(device, torch.bfloat16)
for offset, proj in enumerate(("to_q", "to_k", "to_v")):
expected = fused[offset * hidden : (offset + 1) * hidden]
actual = params[f"transformer_blocks.0.attn.{proj}.weight"]
assert torch.equal(actual, expected), f"{proj} does not match the fused slice"
def test_adaln_modulation_scale_and_shift_are_swapped(comfyui_flux_pipeline):
pipeline, state_dict = comfyui_flux_pipeline
params = dict(pipeline.get_module("transformer").named_parameters())
actual = params["norm_out.linear.weight"]
source = state_dict["final_layer.adaLN_modulation.1.weight"].to(
actual.device, torch.bfloat16
)
half = source.shape[0] // 2
# ComfyUI emits [shift, scale]; AdaLayerNormContinuous expects [scale, shift].
assert torch.equal(actual, torch.cat([source[half:], source[:half]], dim=0))
@@ -0,0 +1,229 @@
# SPDX-License-Identifier: Apache-2.0
import torch
from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import (
bind_comfyui_session,
get_run_state,
initialize_comfyui_pipeline,
release_comfyui_session,
release_run_state,
set_run_state,
)
class _Req:
def __init__(self):
self.extra = {}
self.prompt_embeds = []
self.prompt_seq_lens = None
self.pooled_embeds = []
def test_latent_prep_verify_input_restores_cached_embeds() -> None:
"""Later ComfyUI steps omit embeds; verify_input must restore before checks."""
from types import SimpleNamespace
from sglang.multimodal_gen.runtime.pipelines_core.stages.comfyui_latent_preparation import (
ComfyUILatentPreparationStage,
)
sid = "run-verify"
embeds = [torch.ones(2, 4)]
first = _Req()
first.extra["comfyui_session_id"] = sid
first.prompt_embeds = embeds
first.prompt_seq_lens = [[2]]
bind_comfyui_session(first)
batch = SimpleNamespace(
extra={"comfyui_session_id": sid},
prompt_embeds=[],
prompt=" ",
num_outputs_per_prompt=1,
generator=torch.Generator("cpu"),
num_frames=1,
height=64,
width=64,
latents=None,
prompt_seq_lens=None,
pooled_embeds=None,
negative_prompt_embeds=None,
negative_prompt_seq_lens=None,
neg_pooled_embeds=None,
image_latent=None,
vae_image_sizes=None,
prompt_attention_mask=None,
negative_attention_mask=None,
prompt_embeds_mask=None,
negative_prompt_embeds_mask=None,
)
stage = ComfyUILatentPreparationStage(scheduler=None, transformer=None)
result = stage.verify_input(batch, server_args=None)
assert result.is_valid()
assert torch.equal(batch.prompt_embeds[0], embeds[0])
release_comfyui_session(sid)
def test_session_restores_conditioning_on_later_steps() -> None:
sid = "run-1"
first = _Req()
first.extra["comfyui_session_id"] = sid
first.prompt_embeds = [torch.ones(4, 8)]
first.prompt_seq_lens = [[4]]
bind_comfyui_session(first)
second = _Req()
second.extra["comfyui_session_id"] = sid
bind_comfyui_session(second)
assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0])
assert second.prompt_seq_lens == [[4]]
release_comfyui_session(sid)
third = _Req()
third.extra["comfyui_session_id"] = sid
bind_comfyui_session(third)
assert third.prompt_embeds == []
def test_release_run_state_keeps_conditioning():
sid = "run-keep-cond"
first = _Req()
first.extra = {"comfyui_session_id": sid, "model_payload": {"audio_scale": 0.5}}
first.prompt_embeds = [torch.ones(2, 4)]
bind_comfyui_session(first)
set_run_state(first, {"branch": "alive"})
release_run_state(sid)
assert get_run_state(first) is None
second = _Req()
second.extra = {"comfyui_session_id": sid}
bind_comfyui_session(second)
assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0])
assert second.extra["model_payload"]["audio_scale"] == 0.5
release_comfyui_session(sid)
def test_session_restores_extra_keys_on_later_steps() -> None:
sid = "run-extra"
payload = {"audio_scale": 0.5}
sigmas = torch.tensor([1.0, 0.5, 0.0])
first = _Req()
first.extra = {
"comfyui_session_id": sid,
"model_payload": payload,
"sample_sigmas": sigmas,
}
first.prompt_embeds = [torch.ones(3, 4)]
bind_comfyui_session(first)
second = _Req()
second.extra = {"comfyui_session_id": sid}
bind_comfyui_session(second)
assert second.extra["model_payload"] is payload
assert torch.equal(second.extra["sample_sigmas"], sigmas)
assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0])
release_comfyui_session(sid)
def test_new_run_id_evicts_previous_run_for_same_executor() -> None:
first = _Req()
first.extra = {"comfyui_session_id": "exec1:1", "h3_layout": {"signature": (1, 2)}}
first.prompt_embeds = [torch.ones(2, 4)]
bind_comfyui_session(first)
set_run_state(first, {"branch": "old"})
second = _Req()
second.extra = {"comfyui_session_id": "exec1:2"}
bind_comfyui_session(second)
assert "h3_layout" not in second.extra
assert second.prompt_embeds == []
old = _Req()
old.extra = {"comfyui_session_id": "exec1:1"}
assert get_run_state(old) is None
release_comfyui_session("exec1:2")
def test_fingerprint_mismatch_does_not_restore_extras() -> None:
sid = "exec2:1"
first = _Req()
first.extra = {
"comfyui_session_id": sid,
"comfyui_cache_fp": {"spatial": (5, 15, 27)},
"h3_layout": {"signature": (8, 5, 15, 27, 4)},
}
first.prompt_embeds = [torch.ones(2, 4)]
bind_comfyui_session(first)
second = _Req()
second.extra = {
"comfyui_session_id": sid,
"comfyui_cache_fp": {"spatial": (5, 30, 54)},
}
bind_comfyui_session(second)
assert "h3_layout" not in second.extra
assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0])
release_comfyui_session(sid)
def test_cond_keys_keep_positive_and_negative_apart() -> None:
sid = "exec3:1"
pos = _Req()
pos.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "pos"}
pos.prompt_embeds = [torch.ones(2, 4)]
bind_comfyui_session(pos)
neg = _Req()
neg.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "neg"}
neg.prompt_embeds = [torch.zeros(2, 4)]
bind_comfyui_session(neg)
later_pos = _Req()
later_pos.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "pos"}
bind_comfyui_session(later_pos)
later_neg = _Req()
later_neg.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "neg"}
bind_comfyui_session(later_neg)
assert torch.equal(later_pos.prompt_embeds[0], pos.prompt_embeds[0])
assert torch.equal(later_neg.prompt_embeds[0], neg.prompt_embeds[0])
release_comfyui_session(sid)
class _FakePipeline:
def __init__(self):
self.modules = {}
def _init_vae_geometry(pipeline_config):
from types import SimpleNamespace
initialize_comfyui_pipeline(
_FakePipeline(),
SimpleNamespace(pipeline_config=pipeline_config, comfyui_mode=True),
)
return pipeline_config.vae_config.arch_config
def test_comfyui_mode_derives_flux_vae_scale_factor() -> None:
from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig
arch = _init_vae_geometry(FluxPipelineConfig())
assert arch.vae_scale_factor == 8
def test_comfyui_mode_derives_qwen_vae_scale_factor() -> None:
from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import (
QwenImagePipelineConfig,
)
arch = _init_vae_geometry(QwenImagePipelineConfig())
assert arch.vae_scale_factor == 8
def test_comfyui_mode_h3_vae_post_init_without_latent_stats() -> None:
from sglang.multimodal_gen.configs.pipeline_configs.minimax_h3 import (
MiniMaxH3PipelineConfig,
)
arch = _init_vae_geometry(MiniMaxH3PipelineConfig())
assert arch.latents_mean is None
assert arch.latents_std is None
@@ -92,6 +92,7 @@ class TestComponentLoaderIdentity(unittest.TestCase):
component_paths={},
component_direct_gpu_weight_loading=set(),
resolve_component_attention_backend=lambda *_names: (None, None),
comfyui_mode=False,
)
with patch.object(
@@ -443,6 +443,46 @@ class TestRankLocalSafetensorsRead(unittest.TestCase):
rank_local_checkpoint.tp_local_shape(sources, 0, 2), (4, 4)
)
def test_reads_tp_local_fused_merged_column_per_logical_matrix(self):
with tempfile.TemporaryDirectory() as temp_dir:
file_path = str(Path(temp_dir) / "model.safetensors")
q = torch.arange(16, dtype=torch.bfloat16).reshape(4, 4)
k = torch.arange(16, 32, dtype=torch.bfloat16).reshape(4, 4)
v = torch.arange(32, 48, dtype=torch.bfloat16).reshape(4, 4)
fused = torch.cat((q, k, v), dim=0)
save_file({"qkv": fused}, file_path)
sources = [self._source(file_path, "qkv", (12, 4))]
with safe_open(file_path, framework="pt", device="cpu") as handle:
naive = rank_local_checkpoint.read_tp_local_tensor(
sources,
{file_path: handle},
shard_dim=0,
tp_rank=0,
tp_size=2,
)
rank0 = rank_local_checkpoint.read_tp_local_tensor(
sources,
{file_path: handle},
shard_dim=0,
tp_rank=0,
tp_size=2,
output_sizes=[4, 4, 4],
)
rank1 = rank_local_checkpoint.read_tp_local_tensor(
sources,
{file_path: handle},
shard_dim=0,
tp_rank=1,
tp_size=2,
output_sizes=[4, 4, 4],
)
torch.testing.assert_close(naive, fused[:6])
torch.testing.assert_close(rank0, torch.cat((q[:2], k[:2], v[:2])))
torch.testing.assert_close(rank1, torch.cat((q[2:], k[2:], v[2:])))
self.assertFalse(torch.equal(rank0, naive))
def test_reads_tp_local_merged_row_slice(self):
with tempfile.TemporaryDirectory() as temp_dir:
file_path = str(Path(temp_dir) / "model.safetensors")
@@ -0,0 +1,133 @@
# SPDX-License-Identifier: Apache-2.0
import pickle
from dataclasses import dataclass
import pytest
import torch
from sglang.multimodal_gen.runtime.distributed.ipc_cuda import (
CudaIpcRef,
attach_cuda_tensors,
detach_cuda_tensors,
materialize_cuda_refs,
spill_cuda_tensors,
)
def _require_cuda() -> None:
if not torch.cuda.is_available():
pytest.skip("CUDA is required for CUDA IPC tests")
def test_pickle_copies_cuda_tensor_through_host() -> None:
_require_cuda()
tensor = torch.ones(16, 90, 160, device="cuda", dtype=torch.bfloat16)
assert len(pickle.dumps(tensor)) > tensor.nbytes
def test_ipc_handle_pickle_is_much_smaller_than_tensor() -> None:
_require_cuda()
tensor = torch.ones(16, 90, 160, device="cuda", dtype=torch.bfloat16)
ref = CudaIpcRef.from_tensor(tensor)
assert len(pickle.dumps(ref)) < 2048
rebuilt = ref.materialize()
assert rebuilt.device.type == "cuda"
assert rebuilt.shape == tensor.shape
assert rebuilt.dtype == tensor.dtype
assert torch.equal(rebuilt, tensor)
def test_spill_does_not_mutate_caller_req_tensors() -> None:
_require_cuda()
@dataclass
class _Holder:
latents: torch.Tensor
prompt_embeds: list
latents = torch.arange(24, device="cuda", dtype=torch.float32).reshape(2, 3, 4)
embeds = [torch.ones(3, 5, device="cuda", dtype=torch.float32)]
holder = _Holder(latents=latents, prompt_embeds=embeds)
spilled = spill_cuda_tensors(holder)
assert isinstance(spilled.latents, CudaIpcRef)
assert isinstance(spilled.prompt_embeds[0], CudaIpcRef)
assert holder.latents is latents
assert holder.prompt_embeds[0] is embeds[0]
restored = materialize_cuda_refs(spilled)
assert torch.equal(restored.latents, latents)
assert torch.equal(restored.prompt_embeds[0], embeds[0])
def test_detach_attach_keeps_non_cuda_fields() -> None:
_require_cuda()
@dataclass
class _Holder:
extra: dict
vae_image_sizes: list
image_embeds: list
latents: torch.Tensor
prompt_embeds: list
latents = torch.arange(24, device="cuda", dtype=torch.float32).reshape(2, 3, 4)
embeds = [torch.ones(2, 8, device="cuda", dtype=torch.float32)]
holder = _Holder(
extra={"comfyui_session_id": "run-1"},
vae_image_sizes=[(64, 48)],
image_embeds=[],
latents=latents,
prompt_embeds=embeds,
)
skeleton, tensors = detach_cuda_tensors(holder)
assert holder.latents is latents
assert skeleton.extra["comfyui_session_id"] == "run-1"
assert skeleton.vae_image_sizes == [(64, 48)]
assert skeleton.image_embeds == []
assert skeleton.latents is None
restored = attach_cuda_tensors(skeleton, tensors)
assert restored.extra["comfyui_session_id"] == "run-1"
assert restored.vae_image_sizes == [(64, 48)]
assert restored.image_embeds == []
assert restored.latents.device.type == "cuda"
assert torch.equal(restored.latents, latents)
assert torch.equal(restored.prompt_embeds[0], embeds[0])
def test_cross_process_ipc_roundtrip() -> None:
_require_cuda()
ctx = torch.multiprocessing.get_context("spawn")
tensor = torch.arange(64, device="cuda", dtype=torch.float32).reshape(4, 16)
ref = CudaIpcRef.from_tensor(tensor)
queue = ctx.Queue()
proc = ctx.Process(target=_cross_process_child, args=(queue, pickle.dumps(ref)))
proc.start()
try:
shape, dtype, first, last = queue.get(timeout=30)
finally:
proc.join(timeout=30)
if proc.is_alive():
proc.kill()
proc.join()
assert proc.exitcode == 0
assert shape == (4, 16)
assert dtype == "torch.float32"
assert first == 0.0
assert last == 63.0
def _cross_process_child(queue, payload: bytes) -> None:
ref = pickle.loads(payload)
rebuilt = ref.materialize()
queue.put(
(
tuple(rebuilt.shape),
str(rebuilt.dtype),
float(rebuilt.reshape(-1)[0]),
float(rebuilt.reshape(-1)[-1]),
)
)