[diffusion] feat: add MiniMax-H3 to ComfyUI integrated mode (#35990)
This commit is contained in:
@@ -11,7 +11,7 @@ SGLang diffusion features an end-to-end unified pipeline for accelerating diffus
|
||||
SGLang Diffusion has the following features:
|
||||
- Broad model support: Wan, FastWan, FLUX, Qwen-Image / Qwen-Image 2.1, LongCat-Image, Z-Image, Anima, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3/LTX-2.5, MiniMax-H3, FastH3, VDN-H3, LingBot Video MoE, LingBot World, SANA-Video/SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more
|
||||
- Fast inference speed: empowered by optimized `sgl-kernel` kernels, scheduler/runtime improvements, caching acceleration, and native diffusion hot-path optimizations
|
||||
- Ease of use: OpenAI-compatible api, CLI, and python sdk support
|
||||
- Ease of use: OpenAI-compatible api, CLI, python sdk, and a [ComfyUI plugin](apps/ComfyUI_SGLDiffusion/README.md)
|
||||
- Multi-platform support:
|
||||
- NVIDIA GPUs (H100, H200, A100, B200, 4090, 5090)
|
||||
- AMD GPUs (MI300X, MI325X, MI355X)
|
||||
|
||||
@@ -1,6 +1,14 @@
|
||||
# ComfyUI SGLDiffusion Plugin
|
||||
|
||||
A ComfyUI plugin for integrating with SGLang Diffusion server, supporting image and video generation capabilities.
|
||||
A ComfyUI plugin for SGLang Diffusion. Server mode talks to a standalone HTTP
|
||||
server. Integrated mode keeps ComfyUI's CLIP / VAE / sampler loop and uses
|
||||
SGLang only as a per-step DiT forward.
|
||||
|
||||
Integrated mode no longer ships dedicated `comfyui_*` pipelines. It starts the
|
||||
native Flux / Qwen-Image / Z-Image / MiniMax-H3 pipeline under `--comfyui-mode`,
|
||||
loads a single-file ComfyUI `.safetensors` (or GGUF overlay) through a
|
||||
checkpoint spec, and translates each `apply_model` call through a small
|
||||
per-model adapter.
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -16,7 +24,7 @@ The plugin supports two modes of operation: **Server Mode** (via HTTP API) and *
|
||||
- **Z-Image**: High-speed image generation models (e.g., `Z-Image-Turbo`)
|
||||
- **FLUX**: State-of-the-art text-to-image models (e.g., `FLUX.1-dev`)
|
||||
- **Qwen-Image**: Multi-modal image generation models (e.g., `Qwen-Image`,`Qwen-Image-2512`). *Note: Image editing support is currently experimental and may have some issues.*
|
||||
- **MiniMax-H3**: Joint video-and-audio generation, server mode only (`SGLDiffusion Generate H3`)
|
||||
- **MiniMax-H3**: Joint video-and-audio DiT (`model_type=minimax_h3`). Integrated mode: T2V / I2VA / FL2VA use an `fl2va` checkpoint; R2V needs `ref2va`. CLIP and VAE stay in ComfyUI. Server mode uses `SGLDiffusion Generate MiniMax-H3`.
|
||||
|
||||
### Mode 1: Server Mode (HTTP API)
|
||||
Connect to a standalone SGLang Diffusion server.
|
||||
@@ -26,14 +34,17 @@ Connect to a standalone SGLang Diffusion server.
|
||||
3. **Generate Content**:
|
||||
- `SGLDiffusion Generate Image`: For text-to-image and image editing.
|
||||
- `SGLDiffusion Generate Video`: For text-to-video and image-to-video.
|
||||
- `SGLDiffusion Generate MiniMax-H3`: For joint video-and-audio on a standalone server.
|
||||
4. **LoRA Support**: Use `SGLDiffusion Server Set LoRA` and `SGLDiffusion Server Unset LoRA`.
|
||||
|
||||
### Mode 2: Integrated Mode (Tight Integration)
|
||||
Leverage SGLang's high-performance sampling directly within ComfyUI while using ComfyUI's front-end nodes (CLIP, VAE, etc.).
|
||||
ComfyUI keeps CLIP, VAE, and the sampler loop. SGLang loads only the DiT and
|
||||
runs one forward per sampler step (pass-through scheduler, no text encode /
|
||||
decode on the worker).
|
||||
|
||||
1. **Load Model**: Use the `SGLDiffusion UNET Loader` node to load your diffusion model.
|
||||
2. **Configure Options**: Use the `SGLDiffusion Options` node to set runtime parameters like `num_gpus`, `tp_size`, `model_type`, or `enable_torch_compile`.
|
||||
3. **Sample**: Connect the loaded model to standard ComfyUI samplers. SGLang will handle the sampling process efficiently.
|
||||
3. **Sample**: Connect the loaded model to standard ComfyUI samplers. Each step is packed by a model adapter and sent to the SGLang scheduler.
|
||||
4. **LoRA Support**: Use the `SGLDiffusion LoRA Loader` for native LoRA integration.
|
||||
|
||||
## Adding a Model
|
||||
@@ -74,6 +85,9 @@ Reference workflow files are provided in the `workflows/` directory:
|
||||
- **`z-image_sgld.json`**: High-speed image generation using Z-Image.
|
||||
- **`sgld_text2img.json`**: Server-mode text-to-image generation with LoRA support.
|
||||
- **`sgld_image2video.json`**: Server-mode image-to-video generation.
|
||||
- **`minimax_h3_t2v_sgld.json`**: MiniMax-H3 T2V / I2VA / FL2VA (`fl2va` DiT).
|
||||
- **`minimax_h3_r2v_sgld.json`**: MiniMax-H3 reference-to-video (`ref2va` DiT).
|
||||
- **`minimax_h3_t2v_sgld_upscaler.json`**: H3 two-pass latent upscale (low-res then 3D ×2 refine).
|
||||
|
||||
For other workflows supporting the models, you can easily use SGLang by replacing the official `UNET Loader` node with the `SGLDUNETLoader` node. Similarly, for LoRA support, replace the official LoRA loader with the `SGLDiffusion LoRA Loader`.
|
||||
|
||||
@@ -85,4 +99,45 @@ To use these workflows:
|
||||
|
||||
## Current Implementation
|
||||
|
||||
This plugin provides a high-performance backend for diffusion models in ComfyUI. By leveraging SGLang's optimized kernels and parallelization techniques (Tensor Parallelism, TeaCache, etc.), it significantly accelerates the sampling process, especially for large models like FLUX.
|
||||
SGLang's optimized kernels and parallelism (TP / SP, compile, cache) run on the
|
||||
DiT only. Text encoding and VAE stay in ComfyUI.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph comfy [ComfyUI process]
|
||||
CLIP[CLIP / text encode]
|
||||
VAE[VAE]
|
||||
SAMPLER[Sampler loop]
|
||||
EXEC["SGLDiffusionExecutor"]
|
||||
ADAPT["Model adapter<br/>pack / unpack"]
|
||||
CLIP --> SAMPLER
|
||||
SAMPLER --> EXEC --> ADAPT
|
||||
end
|
||||
|
||||
ADAPT -->|CUDA IPC spill| R0
|
||||
|
||||
subgraph sgl [SGLang]
|
||||
R0[Rank-0 scheduler]
|
||||
R0 -->|comfyui_mode and multi-rank| NCCL["Detach CUDA tensors<br/>NCCL broadcast"]
|
||||
R0 -->|otherwise| PYO["Original SP / CFG / TP<br/>broadcast_pyobj"]
|
||||
NCCL --> PIPE
|
||||
PYO --> PIPE
|
||||
PIPE["Native pipeline<br/>--comfyui-mode"]
|
||||
SPEC["Checkpoint spec<br/>single .safetensors"] --> PIPE
|
||||
PIPE --> STAGE["Latent prep + session cache<br/>+ DenoisingStage"]
|
||||
end
|
||||
|
||||
STAGE -->|noise_pred IPC| EXEC
|
||||
STAGE --> VAE
|
||||
```
|
||||
|
||||
Per sampler step:
|
||||
|
||||
1. The adapter turns ComfyUI `apply_model` tensors into an SGLang `Req`.
|
||||
2. Local ZMQ pickle replaces CUDA tensors with IPC handles so latents stay on GPU.
|
||||
3. Rank 0 materializes the handles. Multi-rank `--comfyui-mode` then detaches CUDA tensors and broadcasts them over NCCL; the general SP / CFG / TP path is still the original whole-list `broadcast_pyobj`.
|
||||
4. The worker pipeline is the native model class with modules trimmed to `transformer` + pass-through scheduler. A single-file checkpoint goes through `comfyui_checkpoints`. After the first step, conditioning stays in a worker session; later steps send latents and the timestep.
|
||||
|
||||
Adding a model means a checkpoint spec plus a `ComfyUIModelAdapter`. There is no extra ComfyUI pipeline class.
|
||||
|
||||
@@ -4,7 +4,6 @@ Provides generator, model patcher, and server API client.
|
||||
"""
|
||||
|
||||
from .generator import SGLDiffusionGenerator
|
||||
from .model_patcher import SGLDModelPatcher
|
||||
from .server_api import SGLDiffusionServerAPI
|
||||
|
||||
__all__ = [
|
||||
@@ -12,3 +11,12 @@ __all__ = [
|
||||
"SGLDModelPatcher",
|
||||
"SGLDiffusionServerAPI",
|
||||
]
|
||||
|
||||
|
||||
def __getattr__(name):
|
||||
# ModelPatcher subclasses ComfyUI and is only needed when a graph loads a DiT.
|
||||
if name == "SGLDModelPatcher":
|
||||
from .model_patcher import SGLDModelPatcher
|
||||
|
||||
return SGLDModelPatcher
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
|
||||
@@ -2,20 +2,103 @@
|
||||
Generator for SGLang Diffusion ComfyUI integration.
|
||||
"""
|
||||
|
||||
import atexit
|
||||
import logging
|
||||
import os
|
||||
|
||||
import psutil
|
||||
from comfy import model_detection, model_management
|
||||
from comfy.utils import (
|
||||
calculate_parameters,
|
||||
load_torch_file,
|
||||
state_dict_prefix_replace,
|
||||
unet_to_diffusers,
|
||||
)
|
||||
from ..executors.flux import FluxExecutor
|
||||
from ..executors.minimax_h3 import MiniMaxH3Executor
|
||||
from ..executors.zimage import ZImageExecutor
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _HeaderTensor:
|
||||
"""Shape-only stand-in so detect_unet_config does not load 66GB weights."""
|
||||
|
||||
def __init__(self, shape):
|
||||
import torch
|
||||
|
||||
self.shape = torch.Size(shape)
|
||||
|
||||
def numel(self):
|
||||
n = 1
|
||||
for dim in self.shape:
|
||||
n *= int(dim)
|
||||
return n
|
||||
|
||||
nelement = numel
|
||||
|
||||
|
||||
def _looks_like_gguf(path: str) -> bool:
|
||||
if not path:
|
||||
return False
|
||||
return path.lower().endswith(".gguf") or (
|
||||
":" in path and path.count("/") == 1 and not os.path.isabs(path)
|
||||
)
|
||||
|
||||
|
||||
def _h3_detect_companion(gguf_ref: str) -> str:
|
||||
"""ComfyUI detect needs H3 safetensors keys; GGUF only overrides the DiT."""
|
||||
name = os.path.basename(gguf_ref).lower()
|
||||
prefer = []
|
||||
if "ref2va" in name:
|
||||
prefer.append("minimax_h3_ref2va_bf16.safetensors")
|
||||
prefer.extend(
|
||||
[
|
||||
"minimax_h3_fl2va_bf16.safetensors",
|
||||
"minimax_h3_ref2va_bf16.safetensors",
|
||||
]
|
||||
)
|
||||
folders = (
|
||||
[os.path.dirname(os.path.abspath(gguf_ref))]
|
||||
if os.path.dirname(gguf_ref)
|
||||
else []
|
||||
)
|
||||
try:
|
||||
import folder_paths
|
||||
|
||||
folders.extend(folder_paths.get_folder_paths("diffusion_models"))
|
||||
except Exception:
|
||||
pass
|
||||
seen: set[str] = set()
|
||||
for folder in folders:
|
||||
if not folder:
|
||||
continue
|
||||
for candidate in prefer:
|
||||
path = os.path.join(folder, candidate)
|
||||
if path in seen:
|
||||
continue
|
||||
seen.add(path)
|
||||
if os.path.isfile(path):
|
||||
return path
|
||||
raise ValueError(
|
||||
"GGUF transformer needs a MiniMax-H3 BF16 safetensors in "
|
||||
"models/diffusion_models for ComfyUI architecture detect "
|
||||
f"(looked for {prefer}). Keep --model-path / unet_name on the "
|
||||
"BF16 file and pass the GGUF via transformer_weights_path."
|
||||
)
|
||||
|
||||
|
||||
def _load_state_dict_for_detection(model_path: str):
|
||||
if _looks_like_gguf(model_path):
|
||||
raise ValueError(
|
||||
f"Cannot detect architecture from GGUF {model_path!r}. "
|
||||
"Use a MiniMax-H3 BF16 safetensors for detect."
|
||||
)
|
||||
if model_path.endswith(".safetensors"):
|
||||
from safetensors import safe_open
|
||||
|
||||
sd = {}
|
||||
with safe_open(model_path, framework="pt", device="cpu") as handle:
|
||||
for key in handle.keys():
|
||||
sd[key] = _HeaderTensor(handle.get_slice(key).get_shape())
|
||||
return sd
|
||||
from comfy.utils import load_torch_file
|
||||
|
||||
return load_torch_file(model_path)
|
||||
|
||||
|
||||
try:
|
||||
from sglang.multimodal_gen import DiffGenerator
|
||||
except ImportError:
|
||||
@@ -23,39 +106,78 @@ except ImportError:
|
||||
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
|
||||
)
|
||||
|
||||
from ..executors import (
|
||||
FluxExecutor,
|
||||
QwenImageEditExecutor,
|
||||
QwenImageExecutor,
|
||||
ZImageExecutor,
|
||||
)
|
||||
from .model_patcher import SGLDModelPatcher
|
||||
|
||||
def _load_executor_classes():
|
||||
"""Qwen adapters import ComfyUI. Keep them optional so CI can load the rest."""
|
||||
classes = [FluxExecutor, ZImageExecutor, MiniMaxH3Executor]
|
||||
try:
|
||||
from ..executors.qwen_image import QwenImageEditExecutor, QwenImageExecutor
|
||||
except ModuleNotFoundError as exc:
|
||||
missing = exc.name or ""
|
||||
if not missing.startswith("comfy"):
|
||||
raise
|
||||
logger.info("ComfyUI is not installed; Qwen image executors are unavailable")
|
||||
else:
|
||||
classes.extend([QwenImageExecutor, QwenImageEditExecutor])
|
||||
return tuple(classes)
|
||||
|
||||
|
||||
_EXECUTOR_CLASSES = _load_executor_classes()
|
||||
|
||||
|
||||
_SHARED = None
|
||||
_ATEXIT_REGISTERED = False
|
||||
|
||||
|
||||
class SGLDiffusionGenerator:
|
||||
"""Generator for SGLang Diffusion models in ComfyUI."""
|
||||
"""Process-wide owner of the one live SGLD worker.
|
||||
|
||||
ComfyUI keeps one ``SGLDUNETLoader`` per node and may reuse a cached
|
||||
``MODEL`` after another graph has already replaced the worker.
|
||||
``sync_scheduler_client`` is also a process singleton, so a second
|
||||
``DiffGenerator`` cannot coexist. All loaders share this object; a
|
||||
stale executor rebinds itself on the next forward.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.model_path = None
|
||||
self.generator = None
|
||||
self.executor = None
|
||||
self.last_options = None
|
||||
self._patcher = None
|
||||
|
||||
self.pipeline_class_dict = {
|
||||
"flux": "ComfyUIFluxPipeline",
|
||||
"lumina2": "ComfyUIZImagePipeline", # zimage
|
||||
"qwen_image": "ComfyUIQwenImagePipeline",
|
||||
"qwen_image_edit": "ComfyUIQwenImageEditPipeline",
|
||||
}
|
||||
self.executor_class_dict = {
|
||||
"flux": FluxExecutor,
|
||||
"lumina2": ZImageExecutor,
|
||||
"qwen_image": QwenImageExecutor,
|
||||
"qwen_image_edit": QwenImageEditExecutor,
|
||||
}
|
||||
# Native pipelines, run under comfyui_mode as a DiT-only forward service.
|
||||
self.pipeline_class_dict = {}
|
||||
self.executor_class_dict = {}
|
||||
for executor_cls in _EXECUTOR_CLASSES:
|
||||
for model_type in executor_cls.adapter_cls.model_types:
|
||||
self.executor_class_dict[model_type] = executor_cls
|
||||
self.pipeline_class_dict[model_type] = (
|
||||
executor_cls.adapter_cls.pipeline_class_name
|
||||
)
|
||||
|
||||
def __del__(self):
|
||||
self.close_generator()
|
||||
@classmethod
|
||||
def shared(cls):
|
||||
global _SHARED, _ATEXIT_REGISTERED
|
||||
if _SHARED is None:
|
||||
_SHARED = cls()
|
||||
if not _ATEXIT_REGISTERED:
|
||||
atexit.register(cls._atexit_close)
|
||||
_ATEXIT_REGISTERED = True
|
||||
return _SHARED
|
||||
|
||||
@classmethod
|
||||
def reset_shared(cls):
|
||||
"""Test helper: drop the process singleton after closing it."""
|
||||
global _SHARED
|
||||
if _SHARED is not None:
|
||||
_SHARED.close_generator()
|
||||
_SHARED = None
|
||||
|
||||
@classmethod
|
||||
def _atexit_close(cls):
|
||||
if _SHARED is not None:
|
||||
_SHARED.close_generator()
|
||||
|
||||
def init_generator(
|
||||
self, model_path: str, pipeline_class_name: str, kwargs: dict = None
|
||||
@@ -66,7 +188,13 @@ class SGLDiffusionGenerator:
|
||||
if kwargs is None:
|
||||
kwargs = {}
|
||||
# Set comfyui_mode for ComfyUI integration
|
||||
kwargs = dict(kwargs)
|
||||
kwargs["comfyui_mode"] = True
|
||||
# ComfyUI already keeps CLIP/VAE in the parent process. Auto image
|
||||
# policy otherwise sets dit_cpu_offload=True and every sampler step
|
||||
# reloads the DiT from CPU.
|
||||
kwargs.setdefault("dit_cpu_offload", False)
|
||||
kwargs = self._server_args_kwargs(kwargs)
|
||||
self.generator = DiffGenerator.from_pretrained(
|
||||
model_path=model_path,
|
||||
pipeline_class_name=pipeline_class_name,
|
||||
@@ -74,62 +202,132 @@ class SGLDiffusionGenerator:
|
||||
)
|
||||
return self.generator
|
||||
|
||||
def kill_generator(self):
|
||||
"""Kill worker processes manually because generator shutdown cannot terminate them."""
|
||||
current_pid = os.getpid()
|
||||
worker_processes = []
|
||||
for proc in psutil.process_iter(["pid", "name", "cmdline"]):
|
||||
try:
|
||||
# Look for sglang-diffusionWorker processes
|
||||
if proc.info["cmdline"]:
|
||||
cmdline = " ".join(proc.info["cmdline"])
|
||||
if "sgl_diffusion::" in cmdline:
|
||||
if proc.info["pid"] != current_pid:
|
||||
worker_processes.append(proc)
|
||||
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||
continue
|
||||
@staticmethod
|
||||
def _server_args_kwargs(kwargs: dict) -> dict:
|
||||
"""Drop plugin-only / stale flags that ServerArgs no longer accepts."""
|
||||
import dataclasses
|
||||
|
||||
if worker_processes:
|
||||
logger.info(
|
||||
f"Found {len(worker_processes)} worker processes to terminate..."
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
|
||||
valid = {f.name for f in dataclasses.fields(ServerArgs)}
|
||||
aliases = {"dp_degree": "dp_size", "cache_strategy": None, "model_type": None}
|
||||
cleaned = {}
|
||||
for key, value in kwargs.items():
|
||||
dest = aliases.get(key, key)
|
||||
if dest is None or dest not in valid:
|
||||
continue
|
||||
cleaned[dest] = value
|
||||
return cleaned
|
||||
|
||||
def _worker_processes(self):
|
||||
gen = self.generator
|
||||
if gen is None:
|
||||
return []
|
||||
return list(getattr(gen, "local_scheduler_process", None) or [])
|
||||
|
||||
def _is_live(self) -> bool:
|
||||
"""True when this owner still has a usable worker + scheduler client."""
|
||||
if self.generator is None:
|
||||
return False
|
||||
try:
|
||||
from sglang.multimodal_gen.runtime.scheduler_client import (
|
||||
sync_scheduler_client,
|
||||
)
|
||||
for proc in worker_processes:
|
||||
try:
|
||||
logger.info(
|
||||
f"Terminating worker process {proc.info['pid']}: {proc.info['name']}"
|
||||
)
|
||||
proc.terminate()
|
||||
proc.wait(timeout=5)
|
||||
except psutil.TimeoutExpired:
|
||||
logger.warning(
|
||||
f"Process {proc.info['pid']} did not terminate, forcing kill..."
|
||||
)
|
||||
try:
|
||||
proc.kill()
|
||||
proc.wait(timeout=2)
|
||||
except (psutil.NoSuchProcess, psutil.TimeoutExpired):
|
||||
pass
|
||||
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||
pass
|
||||
except ImportError:
|
||||
return False
|
||||
ctx = getattr(sync_scheduler_client, "context", None)
|
||||
if ctx is None or getattr(ctx, "closed", False):
|
||||
return False
|
||||
workers = self._worker_processes()
|
||||
if workers and not all(proc.is_alive() for proc in workers):
|
||||
return False
|
||||
return True
|
||||
|
||||
def _can_reuse(self, gather_options: dict) -> bool:
|
||||
return (
|
||||
self.last_options is not None
|
||||
and self.last_options == gather_options
|
||||
and self.generator is not None
|
||||
and getattr(self, "_patcher", None) is not None
|
||||
and self.executor is not None
|
||||
and self._is_live()
|
||||
)
|
||||
|
||||
def _owns_live(self, executor) -> bool:
|
||||
return (
|
||||
executor is not None
|
||||
and self.generator is not None
|
||||
and executor.generator is self.generator
|
||||
and self._is_live()
|
||||
)
|
||||
|
||||
def ensure_executor(self, executor) -> None:
|
||||
"""Rebuild the worker if this cached executor no longer owns it."""
|
||||
if self._owns_live(executor):
|
||||
return
|
||||
reload = getattr(executor, "_sgld_reload", None)
|
||||
if not reload:
|
||||
raise RuntimeError(
|
||||
"SGLD worker is down and this executor has no reload spec"
|
||||
)
|
||||
logger.info(
|
||||
"Rebuilding SGLD worker for %s (previous worker was replaced or died)",
|
||||
reload.get("model_path"),
|
||||
)
|
||||
self.load_model(**reload)
|
||||
executor.generator = self.generator
|
||||
lora = getattr(executor, "_lora_input", None)
|
||||
if lora and lora.get("lora_nickname"):
|
||||
executor.set_lora(**lora)
|
||||
|
||||
def kill_generator(self):
|
||||
"""Force-stop workers this owner started. Do not scan the process table."""
|
||||
for proc in self._worker_processes():
|
||||
if proc is None:
|
||||
continue
|
||||
try:
|
||||
if not proc.is_alive():
|
||||
continue
|
||||
logger.info("Terminating SGLD worker pid=%s", getattr(proc, "pid", "?"))
|
||||
proc.terminate()
|
||||
proc.join(timeout=5)
|
||||
if proc.is_alive():
|
||||
proc.kill()
|
||||
proc.join(timeout=2)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"Failed to terminate SGLD worker %r", proc, exc_info=True
|
||||
)
|
||||
|
||||
def close_generator(self):
|
||||
"""Close and cleanup the generator and all associated resources."""
|
||||
if self.generator is not None:
|
||||
self.generator.shutdown()
|
||||
try:
|
||||
self.generator.shutdown()
|
||||
except Exception:
|
||||
logger.warning("DiffGenerator.shutdown failed", exc_info=True)
|
||||
self.kill_generator()
|
||||
# Clear other references
|
||||
self.last_options = None
|
||||
self.model_path = None
|
||||
self.generator = None
|
||||
self.executor = None
|
||||
self.last_options = None
|
||||
self.model_path = None
|
||||
self.generator = None
|
||||
self.executor = None
|
||||
self._patcher = None
|
||||
|
||||
def get_comfyui_model(self, model_path: str, model_options: dict = None):
|
||||
"""Get ComfyUI model from model path."""
|
||||
if model_options is None:
|
||||
model_options = {}
|
||||
dtype = model_options.get("dtype", None)
|
||||
# Allow loading unets from checkpoint files
|
||||
sd = load_torch_file(model_path)
|
||||
# Header-only read: H3 BF16 is ~66GB and we only need keys + shapes
|
||||
# for detect_unet_config. SGLD loads the real weights later.
|
||||
from comfy import model_detection, model_management
|
||||
from comfy.utils import (
|
||||
calculate_parameters,
|
||||
state_dict_prefix_replace,
|
||||
unet_to_diffusers,
|
||||
)
|
||||
|
||||
sd = _load_state_dict_for_detection(model_path)
|
||||
diffusion_model_prefix = model_detection.unet_prefix_from_state_dict(sd)
|
||||
temp_sd = state_dict_prefix_replace(
|
||||
sd, {diffusion_model_prefix: ""}, filter_keys=True
|
||||
@@ -185,47 +383,78 @@ class SGLDiffusionGenerator:
|
||||
self, model_path: str, model_options: dict = None, sgld_options: dict = None
|
||||
):
|
||||
"""Load model and return model patcher."""
|
||||
gather_options = {
|
||||
raw_sgld_options = dict(sgld_options) if sgld_options else {}
|
||||
reload_kwargs = {
|
||||
"model_path": model_path,
|
||||
"model_options": dict(model_options) if model_options else {},
|
||||
"sgld_options": raw_sgld_options,
|
||||
}
|
||||
sgld_options = dict(raw_sgld_options)
|
||||
model_options = dict(reload_kwargs["model_options"])
|
||||
plugin_flags = {}
|
||||
if "enable_cache_dit" in sgld_options:
|
||||
plugin_flags["enable_cache_dit"] = sgld_options.pop("enable_cache_dit")
|
||||
set_model_type = sgld_options.pop("model_type", None)
|
||||
override = (sgld_options.get("transformer_weights_path") or "").strip()
|
||||
if override:
|
||||
sgld_options["transformer_weights_path"] = override
|
||||
elif _looks_like_gguf(model_path):
|
||||
sgld_options["transformer_weights_path"] = model_path
|
||||
else:
|
||||
sgld_options.pop("transformer_weights_path", None)
|
||||
detect_path = model_path
|
||||
if _looks_like_gguf(model_path):
|
||||
detect_path = _h3_detect_companion(sgld_options["transformer_weights_path"])
|
||||
gather_options = {
|
||||
"model_path": detect_path,
|
||||
"model_options": model_options,
|
||||
"sgld_options": sgld_options,
|
||||
"set_model_type": set_model_type,
|
||||
}
|
||||
if (
|
||||
self.last_options is not None
|
||||
and self.last_options == gather_options
|
||||
and self.generator is not None
|
||||
):
|
||||
return self.generator
|
||||
else:
|
||||
self.close_generator()
|
||||
if self._can_reuse(gather_options):
|
||||
self.executor.enable_cache_dit = plugin_flags.get("enable_cache_dit")
|
||||
return self._patcher
|
||||
|
||||
self.close_generator()
|
||||
self.last_options = gather_options
|
||||
self.model_path = model_path
|
||||
self.model_path = detect_path
|
||||
|
||||
comfyui_model, model_config, model_type = self.get_comfyui_model(
|
||||
model_path, model_options
|
||||
detect_path, model_options
|
||||
)
|
||||
if model_type is None or model_type not in self.pipeline_class_dict:
|
||||
raise ValueError(f"Unsupported model type: {model_type}")
|
||||
|
||||
set_model_type = sgld_options.pop("model_type", None) if sgld_options else None
|
||||
if set_model_type is not None and set_model_type in self.pipeline_class_dict:
|
||||
model_type = set_model_type
|
||||
|
||||
pipeline_class_name = self.pipeline_class_dict[model_type]
|
||||
self.generator = self.init_generator(
|
||||
model_path, pipeline_class_name, sgld_options
|
||||
detect_path, pipeline_class_name, sgld_options
|
||||
)
|
||||
|
||||
executor_class = self.executor_class_dict[model_type]
|
||||
self.executor = executor_class(
|
||||
self.generator, model_path, comfyui_model, model_config
|
||||
self.generator, detect_path, comfyui_model, model_config
|
||||
)
|
||||
self.executor.enable_cache_dit = plugin_flags.get("enable_cache_dit")
|
||||
self.executor._sgld_reload = reload_kwargs
|
||||
self.executor._ensure_runtime = self.ensure_executor
|
||||
comfyui_model.diffusion_model = self.executor
|
||||
|
||||
from comfy import model_management
|
||||
from comfy.patcher_extension import WrappersMP
|
||||
|
||||
from .model_patcher import SGLDModelPatcher
|
||||
|
||||
load_device = model_management.get_torch_device()
|
||||
offload_device = model_management.unet_offload_device()
|
||||
|
||||
return SGLDModelPatcher(
|
||||
self._patcher = SGLDModelPatcher(
|
||||
comfyui_model, load_device, offload_device, model_type=model_type
|
||||
)
|
||||
self._patcher.add_wrapper_with_key(
|
||||
WrappersMP.SAMPLER_SAMPLE,
|
||||
"sgld_session",
|
||||
self.executor.sampler_sample_wrapper,
|
||||
)
|
||||
return self._patcher
|
||||
|
||||
@@ -48,6 +48,17 @@ class SGLDModelPatcher(ModelPatcher):
|
||||
n.backup = self.backup
|
||||
n.object_patches_backup = self.object_patches_backup
|
||||
n.lora_cache = copy.copy(self.lora_cache)
|
||||
n.model_type = self.model_type
|
||||
n.wrappers = {}
|
||||
for wrapper_type, keyed in self.wrappers.items():
|
||||
n.wrappers[wrapper_type] = {
|
||||
key: wrappers.copy() for key, wrappers in keyed.items()
|
||||
}
|
||||
n.callbacks = {}
|
||||
for call_type, keyed in self.callbacks.items():
|
||||
n.callbacks[call_type] = {
|
||||
key: callbacks.copy() for key, callbacks in keyed.items()
|
||||
}
|
||||
return n
|
||||
|
||||
def model_size(self):
|
||||
|
||||
@@ -3,15 +3,50 @@ ComfyUI SGLang Diffusion executors package.
|
||||
Provides executor classes for different model types.
|
||||
"""
|
||||
|
||||
from .adapter import ComfyUIModelAdapter, PackedForward, get_adapter_class
|
||||
from .base import SGLDiffusionExecutor
|
||||
from .flux import FluxExecutor
|
||||
from .qwen_image import QwenImageEditExecutor, QwenImageExecutor
|
||||
from .zimage import ZImageExecutor
|
||||
from .flux import FluxAdapter, FluxExecutor
|
||||
from .minimax_h3 import MiniMaxH3Adapter, MiniMaxH3Executor
|
||||
from .zimage import ZImageAdapter, ZImageExecutor
|
||||
|
||||
# Qwen adapters import ComfyUI (`comfy.ldm.common_dit`). Keep that optional so
|
||||
# unit tests and SGLD-only paths can load Flux / Z-Image without ComfyUI.
|
||||
|
||||
__all__ = [
|
||||
"ComfyUIModelAdapter",
|
||||
"PackedForward",
|
||||
"SGLDiffusionExecutor",
|
||||
"FluxAdapter",
|
||||
"FluxExecutor",
|
||||
"MiniMaxH3Adapter",
|
||||
"MiniMaxH3Executor",
|
||||
"ZImageAdapter",
|
||||
"ZImageExecutor",
|
||||
"QwenImageExecutor",
|
||||
"QwenImageEditExecutor",
|
||||
"get_adapter_class",
|
||||
]
|
||||
|
||||
|
||||
def __getattr__(name):
|
||||
if name in {
|
||||
"QwenImageExecutor",
|
||||
"QwenImageEditExecutor",
|
||||
"QwenImageAdapter",
|
||||
"QwenImageEditAdapter",
|
||||
}:
|
||||
from .qwen_image import (
|
||||
QwenImageAdapter,
|
||||
QwenImageEditAdapter,
|
||||
QwenImageEditExecutor,
|
||||
QwenImageExecutor,
|
||||
)
|
||||
|
||||
mapping = {
|
||||
"QwenImageAdapter": QwenImageAdapter,
|
||||
"QwenImageEditAdapter": QwenImageEditAdapter,
|
||||
"QwenImageExecutor": QwenImageExecutor,
|
||||
"QwenImageEditExecutor": QwenImageEditExecutor,
|
||||
}
|
||||
return mapping[name]
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Per-model adapters for the ComfyUI DiT-forward contract.
|
||||
|
||||
The shared executor owns request construction and the ZMQ round-trip.
|
||||
Each adapter only translates between ComfyUI's ``apply_model`` tensors and
|
||||
the fields SGLang's ``Req`` expects. Design the interface around H3 (nested
|
||||
latents, structured payload), not Flux's three-tensor case.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
_ADAPTERS: dict[str, type[ComfyUIModelAdapter]] = {}
|
||||
|
||||
|
||||
@dataclass
|
||||
class PackedForward:
|
||||
"""One ComfyUI sampler step, already translated into SGLang tensors."""
|
||||
|
||||
latents: torch.Tensor
|
||||
timesteps: torch.Tensor
|
||||
prompt_embeds: list[torch.Tensor]
|
||||
height: int
|
||||
width: int
|
||||
guidance_scale: float = 1.0
|
||||
prompt_seq_lens: list[list[int]] | None = None
|
||||
pooled_embeds: list[torch.Tensor] | None = None
|
||||
extra_req: dict[str, Any] = field(default_factory=dict)
|
||||
unpack_ctx: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class ComfyUIModelAdapter:
|
||||
"""Model-specific pack / unpack / fill_req for one ComfyUI DiT family."""
|
||||
|
||||
model_types: tuple[str, ...] = ()
|
||||
pipeline_class_name: str = ""
|
||||
|
||||
def __init_subclass__(cls, **kwargs):
|
||||
super().__init_subclass__(**kwargs)
|
||||
for model_type in cls.model_types:
|
||||
_ADAPTERS[model_type] = cls
|
||||
|
||||
def pack(
|
||||
self, x: torch.Tensor, timestep: torch.Tensor, context, **kwargs
|
||||
) -> PackedForward:
|
||||
raise NotImplementedError
|
||||
|
||||
def unpack(
|
||||
self, noise_pred: torch.Tensor, packed: PackedForward, x: torch.Tensor
|
||||
) -> torch.Tensor:
|
||||
return noise_pred.to(x.device)
|
||||
|
||||
def fill_req(self, req, packed: PackedForward) -> None:
|
||||
req.latents = packed.latents
|
||||
req.timesteps = packed.timesteps
|
||||
req.prompt_embeds = packed.prompt_embeds
|
||||
req.raw_latent_shape = torch.tensor(packed.latents.shape, dtype=torch.long)
|
||||
req.do_classifier_free_guidance = False
|
||||
if packed.prompt_seq_lens is not None:
|
||||
req.prompt_seq_lens = packed.prompt_seq_lens
|
||||
if packed.pooled_embeds is not None:
|
||||
req.pooled_embeds = packed.pooled_embeds
|
||||
for key, value in packed.extra_req.items():
|
||||
setattr(req, key, value)
|
||||
|
||||
def drop_cached_fields(self, packed: PackedForward) -> None:
|
||||
packed.prompt_embeds = []
|
||||
packed.prompt_seq_lens = None
|
||||
packed.pooled_embeds = None
|
||||
packed.extra_req.pop("image_latent", None)
|
||||
|
||||
|
||||
def get_adapter_class(model_type: str) -> type[ComfyUIModelAdapter]:
|
||||
if model_type not in _ADAPTERS:
|
||||
raise ValueError(
|
||||
f"Unsupported ComfyUI model type {model_type!r}. "
|
||||
f"Registered: {sorted(_ADAPTERS)}"
|
||||
)
|
||||
return _ADAPTERS[model_type]
|
||||
|
||||
|
||||
def registered_model_types() -> list[str]:
|
||||
return sorted(_ADAPTERS)
|
||||
@@ -2,11 +2,23 @@
|
||||
Base executor class for SGLang Diffusion ComfyUI integration.
|
||||
"""
|
||||
|
||||
import uuid
|
||||
|
||||
import torch
|
||||
|
||||
try:
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
|
||||
from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
|
||||
except ImportError:
|
||||
print(
|
||||
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
|
||||
)
|
||||
|
||||
|
||||
class SGLDiffusionExecutor(torch.nn.Module):
|
||||
"""Base executor class for SGLang Diffusion models in ComfyUI."""
|
||||
"""Shared ComfyUI DiT-forward executor. Per-model logic lives on the adapter."""
|
||||
|
||||
adapter_cls = None
|
||||
|
||||
def __init__(self, generator, model_path, model, config):
|
||||
super(SGLDiffusionExecutor, self).__init__()
|
||||
@@ -16,6 +28,15 @@ class SGLDiffusionExecutor(torch.nn.Module):
|
||||
self.dtype = config.unet_config["dtype"]
|
||||
self.config = config
|
||||
self.loras = []
|
||||
self._lora_input = None
|
||||
self._sgld_reload = None
|
||||
self._ensure_runtime = None
|
||||
if self.adapter_cls is None:
|
||||
raise TypeError(f"{type(self).__name__} must set adapter_cls")
|
||||
self.adapter = self.adapter_cls()
|
||||
self.session_id = uuid.uuid4().hex
|
||||
self._run_id = 0
|
||||
self._sent_conds: set[tuple] = set()
|
||||
|
||||
@staticmethod
|
||||
def should_suppress_logs(timestep):
|
||||
@@ -26,7 +47,13 @@ class SGLDiffusionExecutor(torch.nn.Module):
|
||||
|
||||
def set_lora(self, lora_nickname=None, lora_path=None, strength=None, target=None):
|
||||
"""Set LoRA adapter using SGLang Diffusion API."""
|
||||
if len(lora_nickname) > 0:
|
||||
self._lora_input = {
|
||||
"lora_nickname": lora_nickname,
|
||||
"lora_path": lora_path,
|
||||
"strength": strength,
|
||||
"target": target,
|
||||
}
|
||||
if lora_nickname and len(lora_nickname) > 0:
|
||||
self.generator.set_lora(
|
||||
lora_nickname=lora_nickname,
|
||||
lora_path=lora_path,
|
||||
@@ -34,23 +61,87 @@ class SGLDiffusionExecutor(torch.nn.Module):
|
||||
target=target,
|
||||
)
|
||||
|
||||
def _unpack_latents(self, latents, height, width, channels):
|
||||
"""Unpack latents from packed format to standard format."""
|
||||
batch_size = latents.shape[0]
|
||||
latents = latents.view(batch_size, height // 2, width // 2, channels, 2, 2)
|
||||
latents = latents.permute(0, 3, 1, 4, 2, 5)
|
||||
latents = latents.reshape(batch_size, channels, height, width)
|
||||
def begin_sampler_run(self) -> None:
|
||||
"""One ComfyUI ``sampler.sample()`` invocation is one cache lifetime."""
|
||||
self._run_id += 1
|
||||
self._sent_conds = set()
|
||||
|
||||
return latents
|
||||
def end_sampler_run(self) -> None:
|
||||
"""Run cache is evicted on the next bind of a newer id for this executor."""
|
||||
|
||||
def _pack_latents(self, latents):
|
||||
"""Pack latents from standard format to packed format."""
|
||||
batch_size, num_channels_latents, height, width = latents.shape
|
||||
latents = latents.view(
|
||||
batch_size, num_channels_latents, height // 2, 2, width // 2, 2
|
||||
def sampler_sample_wrapper(self, executor, *args, **kwargs):
|
||||
self.begin_sampler_run()
|
||||
try:
|
||||
return executor(*args, **kwargs)
|
||||
finally:
|
||||
self.end_sampler_run()
|
||||
|
||||
def comfyui_session_id(self) -> str:
|
||||
return f"{self.session_id}:{self._run_id}"
|
||||
|
||||
def _cond_key(self, packed) -> tuple | None:
|
||||
embeds = packed.prompt_embeds
|
||||
if not embeds:
|
||||
return None
|
||||
tensor = embeds[0]
|
||||
if not torch.is_tensor(tensor) or tensor.numel() == 0:
|
||||
return None
|
||||
flat = tensor.reshape(-1)
|
||||
return (
|
||||
tuple(int(dim) for dim in tensor.shape),
|
||||
float(flat[0].item()),
|
||||
float(flat[-1].item()),
|
||||
)
|
||||
latents = latents.permute(0, 2, 4, 1, 3, 5)
|
||||
latents = latents.reshape(
|
||||
batch_size, (height // 2) * (width // 2), num_channels_latents * 4
|
||||
|
||||
def _mark_and_maybe_drop(self, packed) -> None:
|
||||
key = self._cond_key(packed)
|
||||
if key is not None:
|
||||
packed.extra_req["comfyui_cond_key"] = repr(key)
|
||||
if key in self._sent_conds:
|
||||
self.adapter.drop_cached_fields(packed)
|
||||
else:
|
||||
self._sent_conds.add(key)
|
||||
|
||||
def _sampling_params_kwargs(self, packed, timestep) -> dict:
|
||||
return {
|
||||
"prompt": " ",
|
||||
"guidance_scale": packed.guidance_scale,
|
||||
"height": packed.height,
|
||||
"width": packed.width,
|
||||
"num_frames": 1,
|
||||
"num_inference_steps": 1,
|
||||
"save_output": False,
|
||||
"suppress_logs": self.should_suppress_logs(timestep),
|
||||
}
|
||||
|
||||
def _execute_packed(self, packed, x, timestep):
|
||||
ensure = getattr(self, "_ensure_runtime", None)
|
||||
if ensure is not None:
|
||||
ensure(self)
|
||||
self._mark_and_maybe_drop(packed)
|
||||
sampling_params = SamplingParams.from_user_sampling_params_args(
|
||||
self.model_path,
|
||||
server_args=self.generator.server_args,
|
||||
**self._sampling_params_kwargs(packed, timestep),
|
||||
)
|
||||
return latents
|
||||
req = prepare_request(
|
||||
server_args=self.generator.server_args,
|
||||
sampling_params=sampling_params,
|
||||
)
|
||||
self.adapter.fill_req(req, packed)
|
||||
extra = dict(req.extra or {})
|
||||
extra["comfyui_session_id"] = self.comfyui_session_id()
|
||||
for key in ("comfyui_cond_key", "comfyui_cache_fp"):
|
||||
value = packed.extra_req.get(key)
|
||||
if value is not None:
|
||||
extra[key] = value
|
||||
req.extra = extra
|
||||
req.generator = [
|
||||
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
|
||||
]
|
||||
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
|
||||
return self.adapter.unpack(output_batch.noise_pred, packed, x)
|
||||
|
||||
def forward(self, x, timestep, context, **kwargs):
|
||||
packed = self.adapter.pack(x, timestep, context, **kwargs)
|
||||
return self._execute_packed(packed, x, timestep)
|
||||
|
||||
@@ -1,69 +1,69 @@
|
||||
"""
|
||||
Flux executor for SGLang Diffusion ComfyUI integration.
|
||||
"""
|
||||
"""Flux adapter for the ComfyUI DiT-forward contract."""
|
||||
|
||||
import torch
|
||||
|
||||
try:
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
|
||||
from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
|
||||
except ImportError:
|
||||
print(
|
||||
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
|
||||
)
|
||||
|
||||
from .adapter import ComfyUIModelAdapter, PackedForward
|
||||
from .base import SGLDiffusionExecutor
|
||||
|
||||
|
||||
def _flux_guidance_scale(guidance) -> float:
|
||||
if guidance is None:
|
||||
return 3.5
|
||||
if torch.is_tensor(guidance):
|
||||
return float(guidance.detach().reshape(-1)[0].item())
|
||||
return float(guidance)
|
||||
|
||||
|
||||
class FluxAdapter(ComfyUIModelAdapter):
|
||||
model_types = ("flux",)
|
||||
pipeline_class_name = "FluxPipeline"
|
||||
|
||||
def pack(
|
||||
self, x, timestep, context, y=None, guidance=None, **kwargs
|
||||
) -> PackedForward:
|
||||
packed = self._pack_latents(x)
|
||||
t5_seq = int(context.shape[-2]) if context.ndim >= 2 else int(context.shape[0])
|
||||
clip_batch = int(y.shape[0]) if y is not None else 1
|
||||
return PackedForward(
|
||||
latents=packed,
|
||||
timesteps=timestep * 1000.0,
|
||||
prompt_embeds=[y, context],
|
||||
prompt_seq_lens=[[clip_batch], [t5_seq]],
|
||||
pooled_embeds=[y],
|
||||
height=x.shape[-2] * 8,
|
||||
width=x.shape[-1] * 8,
|
||||
guidance_scale=_flux_guidance_scale(guidance),
|
||||
unpack_ctx={
|
||||
"height": x.shape[-2],
|
||||
"width": x.shape[-1],
|
||||
"channels": x.shape[1],
|
||||
},
|
||||
)
|
||||
|
||||
def unpack(self, noise_pred, packed, x):
|
||||
ctx = packed.unpack_ctx
|
||||
return self._unpack_latents(
|
||||
noise_pred, ctx["height"], ctx["width"], ctx["channels"]
|
||||
).to(x.device)
|
||||
|
||||
@staticmethod
|
||||
def _unpack_latents(latents, height, width, channels):
|
||||
batch_size = latents.shape[0]
|
||||
latents = latents.view(batch_size, height // 2, width // 2, channels, 2, 2)
|
||||
latents = latents.permute(0, 3, 1, 4, 2, 5)
|
||||
return latents.reshape(batch_size, channels, height, width)
|
||||
|
||||
@staticmethod
|
||||
def _pack_latents(latents):
|
||||
batch_size, num_channels_latents, height, width = latents.shape
|
||||
latents = latents.view(
|
||||
batch_size, num_channels_latents, height // 2, 2, width // 2, 2
|
||||
)
|
||||
latents = latents.permute(0, 2, 4, 1, 3, 5)
|
||||
return latents.reshape(
|
||||
batch_size, (height // 2) * (width // 2), num_channels_latents * 4
|
||||
)
|
||||
|
||||
|
||||
class FluxExecutor(SGLDiffusionExecutor):
|
||||
"""Executor for Flux models in ComfyUI."""
|
||||
|
||||
def __init__(self, generator, model_path, model, config):
|
||||
super().__init__(generator, model_path, model, config)
|
||||
|
||||
def forward(self, x, timestep, context, y=None, guidance=None, **kwargs):
|
||||
"""Forward pass for Flux model."""
|
||||
hidden_states = self._pack_latents(x)
|
||||
timesteps = timestep * 1000.0
|
||||
encoder_hidden_states = context
|
||||
pooled_projections = y
|
||||
guidance = guidance * 1000.0
|
||||
|
||||
B, C, H, W = x.shape
|
||||
height = H * 8
|
||||
width = W * 8
|
||||
# Create SamplingParams
|
||||
sampling_params = SamplingParams.from_user_sampling_params_args(
|
||||
self.model_path,
|
||||
server_args=self.generator.server_args,
|
||||
prompt=" ",
|
||||
guidance_scale=3.5, # Flux typically uses embedded_cfg_scale=3.5
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=1,
|
||||
num_inference_steps=1,
|
||||
save_output=False,
|
||||
suppress_logs=self.should_suppress_logs(timestep),
|
||||
)
|
||||
|
||||
# Prepare request (converts SamplingParams to Req)
|
||||
req = prepare_request(
|
||||
server_args=self.generator.server_args,
|
||||
sampling_params=sampling_params,
|
||||
)
|
||||
req.latents = hidden_states # Set as [B, S, D] format directly
|
||||
req.timesteps = timesteps # ComfyUI's timesteps parameter
|
||||
req.prompt_embeds = [pooled_projections, encoder_hidden_states] # [CLIP, T5]
|
||||
req.raw_latent_shape = torch.tensor(hidden_states.shape, dtype=torch.long)
|
||||
|
||||
# Set pooled_projections (required by Flux)
|
||||
req.pooled_embeds = [pooled_projections] # List format as per Req definition
|
||||
req.do_classifier_free_guidance = False
|
||||
req.generator = [
|
||||
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
|
||||
]
|
||||
|
||||
# Send request to scheduler
|
||||
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
|
||||
noise_pred = output_batch.noise_pred
|
||||
return self._unpack_latents(noise_pred, H, W, C).to(x.device)
|
||||
adapter_cls = FluxAdapter
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""MiniMax-H3 adapter for the ComfyUI DiT-forward contract."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import fields
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.configs.sample.minimax_h3 import MiniMaxH3SamplingParams
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import (
|
||||
DEFAULT_SIGMA_SHIFT_AUDIO,
|
||||
DEFAULT_SIGMA_SHIFT_VIDEO,
|
||||
serialize_comfyui_layout,
|
||||
time_shift_sigma,
|
||||
time_shift_slope,
|
||||
)
|
||||
|
||||
from .adapter import ComfyUIModelAdapter, PackedForward
|
||||
from .base import SGLDiffusionExecutor
|
||||
|
||||
# H3 pins loop/CFG fields with init=False; ComfyUI owns those.
|
||||
_H3_PINNED_SAMPLING_FIELDS = frozenset(
|
||||
f.name for f in fields(MiniMaxH3SamplingParams) if not f.init
|
||||
)
|
||||
|
||||
|
||||
def drop_h3_pinned_sampling_fields(kwargs: dict[str, Any]) -> dict[str, Any]:
|
||||
for name in _H3_PINNED_SAMPLING_FIELDS:
|
||||
kwargs.pop(name, None)
|
||||
return kwargs
|
||||
|
||||
|
||||
_WORKER_TRANSFORMER_OPTION_KEYS = (
|
||||
"minimax_h3_sigma_shift_video",
|
||||
"minimax_h3_sigma_shift_audio",
|
||||
"sample_sigmas",
|
||||
)
|
||||
|
||||
|
||||
def worker_transformer_options(opts: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""Keep only pickle-safe H3 keys; drop ComfyUI model_sampling etc."""
|
||||
src = opts or {}
|
||||
return {key: src[key] for key in _WORKER_TRANSFORMER_OPTION_KEYS if key in src}
|
||||
|
||||
|
||||
def _split_av(x) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
if isinstance(x, (list, tuple)) and len(x) >= 2:
|
||||
return x[0], x[1]
|
||||
unbind = getattr(x, "unbind", None)
|
||||
if callable(unbind):
|
||||
parts = unbind()
|
||||
if len(parts) >= 2:
|
||||
return parts[0], parts[1]
|
||||
raise TypeError(
|
||||
f"MiniMax H3 ComfyUI forward expects x to be [video, audio], got {type(x)!r}"
|
||||
)
|
||||
|
||||
|
||||
def _as_context(context: torch.Tensor) -> torch.Tensor:
|
||||
if context.ndim == 3:
|
||||
return context[0]
|
||||
return context
|
||||
|
||||
|
||||
def _cache_fingerprint(video, sample_sigmas, serialized_layout) -> dict[str, Any]:
|
||||
spatial = tuple(int(dim) for dim in video.shape[-3:])
|
||||
sigmas_fp = None
|
||||
if sample_sigmas is not None:
|
||||
values = (
|
||||
sample_sigmas.reshape(-1)
|
||||
if torch.is_tensor(sample_sigmas)
|
||||
else sample_sigmas
|
||||
)
|
||||
sigmas_fp = (float(values[0]), float(values[-1]), int(len(values)))
|
||||
layout_sig = None
|
||||
if isinstance(serialized_layout, dict):
|
||||
layout_sig = serialized_layout.get("signature")
|
||||
return {"spatial": spatial, "sigmas": sigmas_fp, "layout": layout_sig}
|
||||
|
||||
|
||||
class MiniMaxH3Adapter(ComfyUIModelAdapter):
|
||||
model_types = ("minimax_h3",)
|
||||
pipeline_class_name = "MiniMaxH3Pipeline"
|
||||
|
||||
def pack(self, x, timestep, context, **kwargs) -> PackedForward:
|
||||
video, audio_src = _split_av(x)
|
||||
# PackedLayout is a ComfyUI class; the SGLD worker cannot unpickle it.
|
||||
payload = dict(kwargs.get("minimax_payload") or {})
|
||||
layout = payload.pop("layout", None)
|
||||
transformer_options = kwargs.get("transformer_options") or {}
|
||||
scale = float(payload.get("audio_scale", 1.0))
|
||||
shift_v = float(
|
||||
transformer_options.get(
|
||||
"minimax_h3_sigma_shift_video", DEFAULT_SIGMA_SHIFT_VIDEO
|
||||
)
|
||||
)
|
||||
shift_a = float(
|
||||
transformer_options.get(
|
||||
"minimax_h3_sigma_shift_audio", DEFAULT_SIGMA_SHIFT_AUDIO
|
||||
)
|
||||
)
|
||||
sigma_v = (timestep.reshape(-1)[0] / 1000.0).to(torch.float32).clamp(min=1e-6)
|
||||
sigma_a = time_shift_sigma(sigma_v, shift_v, shift_a)
|
||||
carry = (sigma_a / sigma_v).to(audio_src.dtype)
|
||||
audio = audio_src * carry if scale != 1.0 else audio_src
|
||||
text = _as_context(context)
|
||||
sample_sigmas = transformer_options.get("sample_sigmas")
|
||||
extra_req: dict[str, Any] = {
|
||||
"audio_latents": audio,
|
||||
"h3_payload": payload,
|
||||
"h3_transformer_options": worker_transformer_options(transformer_options),
|
||||
"h3_sample_sigmas": sample_sigmas,
|
||||
"h3_context": text,
|
||||
}
|
||||
serialized = serialize_comfyui_layout(layout)
|
||||
if serialized is not None:
|
||||
extra_req["h3_layout"] = serialized
|
||||
for name in ("denoise_mask", "audio_denoise_mask"):
|
||||
mask = kwargs.get(name)
|
||||
if mask is not None:
|
||||
extra_req[f"h3_{name}"] = mask
|
||||
if sample_sigmas is not None:
|
||||
extra_req["sigmas"] = sample_sigmas
|
||||
extra_req["comfyui_cache_fp"] = _cache_fingerprint(
|
||||
video, sample_sigmas, serialized
|
||||
)
|
||||
return PackedForward(
|
||||
latents=video,
|
||||
timesteps=timestep,
|
||||
prompt_embeds=[text],
|
||||
prompt_seq_lens=[[int(text.shape[0])]],
|
||||
height=int(video.shape[-2]),
|
||||
width=int(video.shape[-1]),
|
||||
extra_req=extra_req,
|
||||
unpack_ctx={
|
||||
"audio_scale": scale,
|
||||
"audio_src": audio_src,
|
||||
"carry": carry,
|
||||
"sigma_v": sigma_v,
|
||||
"sigma_a": sigma_a,
|
||||
"shift_v": shift_v,
|
||||
"shift_a": shift_a,
|
||||
},
|
||||
)
|
||||
|
||||
def unpack(self, noise_pred, packed, x):
|
||||
video_x, audio_x = _split_av(x)
|
||||
if isinstance(noise_pred, (list, tuple)):
|
||||
v_video, v_audio = noise_pred[0], noise_pred[1]
|
||||
else:
|
||||
raise TypeError(
|
||||
"MiniMax H3 unpack expects [video, audio] noise_pred, "
|
||||
f"got {type(noise_pred)!r}"
|
||||
)
|
||||
# Match ComfyUI MiniMaxH3._forward: negated velocities, and audio
|
||||
# scaled by d(sigma_a)/d(sigma_v) so the sampler's sigma_v ODE is
|
||||
# the audio stream's true ODE on its shifted schedule.
|
||||
v_video = (-v_video).to(device=video_x.device, dtype=video_x.dtype)
|
||||
v_audio = (-v_audio).to(device=audio_x.device, dtype=audio_x.dtype)
|
||||
ctx = packed.unpack_ctx
|
||||
scale = float(ctx.get("audio_scale", 1.0))
|
||||
if scale != 1.0:
|
||||
audio_src = ctx["audio_src"]
|
||||
carry = ctx["carry"]
|
||||
sigma_a = ctx["sigma_a"]
|
||||
v_audio = (1.0 - scale) * (audio_src * carry) + (
|
||||
1.0 + (scale - 1.0) * sigma_a
|
||||
).to(v_audio.dtype) * v_audio
|
||||
slope_a = time_shift_slope(
|
||||
ctx["sigma_v"],
|
||||
ctx.get("shift_v", DEFAULT_SIGMA_SHIFT_VIDEO),
|
||||
ctx.get("shift_a", DEFAULT_SIGMA_SHIFT_AUDIO),
|
||||
).to(device=v_audio.device, dtype=v_audio.dtype)
|
||||
v_audio = slope_a * v_audio
|
||||
return [v_video, v_audio]
|
||||
|
||||
def fill_req(self, req, packed: PackedForward) -> None:
|
||||
super().fill_req(req, packed)
|
||||
extra = dict(req.extra or {})
|
||||
for key in (
|
||||
"h3_payload",
|
||||
"h3_transformer_options",
|
||||
"h3_sample_sigmas",
|
||||
"h3_context",
|
||||
"h3_layout",
|
||||
"h3_denoise_mask",
|
||||
"h3_audio_denoise_mask",
|
||||
"comfyui_cache_fp",
|
||||
"comfyui_cond_key",
|
||||
):
|
||||
value = packed.extra_req.get(key)
|
||||
if value is not None:
|
||||
extra[key] = value
|
||||
req.extra = extra
|
||||
|
||||
def drop_cached_fields(self, packed: PackedForward) -> None:
|
||||
super().drop_cached_fields(packed)
|
||||
packed.extra_req.pop("h3_payload", None)
|
||||
packed.extra_req.pop("h3_transformer_options", None)
|
||||
packed.extra_req.pop("h3_sample_sigmas", None)
|
||||
packed.extra_req.pop("h3_context", None)
|
||||
packed.extra_req.pop("h3_layout", None)
|
||||
packed.extra_req.pop("h3_denoise_mask", None)
|
||||
packed.extra_req.pop("h3_audio_denoise_mask", None)
|
||||
packed.extra_req.pop("sigmas", None)
|
||||
# Keep comfyui_cache_fp / comfyui_cond_key; later steps still send them.
|
||||
|
||||
|
||||
class MiniMaxH3Executor(SGLDiffusionExecutor):
|
||||
adapter_cls = MiniMaxH3Adapter
|
||||
# ComfyUI MiniMaxH3.extra_conds reads this when a denoise_mask is present.
|
||||
patch_size = (1, 2, 2)
|
||||
|
||||
def preprocess_text_embeds(self, text_states):
|
||||
"""ComfyUI extra_conds runs this once per sample.
|
||||
|
||||
Token refinement happens on the SGLD worker (the DiT lives there).
|
||||
Identity-pass here so PackedLayout still sees the correct token length.
|
||||
"""
|
||||
return text_states
|
||||
|
||||
def _sampling_params_kwargs(self, packed, timestep) -> dict:
|
||||
kwargs = super()._sampling_params_kwargs(packed, timestep)
|
||||
drop_h3_pinned_sampling_fields(kwargs)
|
||||
enable_cache_dit = getattr(self, "enable_cache_dit", None)
|
||||
if enable_cache_dit is not None:
|
||||
kwargs["enable_cache_dit"] = bool(enable_cache_dit)
|
||||
return kwargs
|
||||
@@ -1,31 +1,40 @@
|
||||
"""
|
||||
QwenImage executor for SGLang Diffusion ComfyUI integration.
|
||||
"""
|
||||
|
||||
import torch
|
||||
|
||||
try:
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
|
||||
from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
|
||||
except ImportError:
|
||||
print(
|
||||
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
|
||||
)
|
||||
"""Qwen-Image adapters for the ComfyUI DiT-forward contract."""
|
||||
|
||||
import comfy.ldm.common_dit
|
||||
|
||||
from .adapter import ComfyUIModelAdapter, PackedForward
|
||||
from .base import SGLDiffusionExecutor
|
||||
|
||||
|
||||
class QwenImageExecutor(SGLDiffusionExecutor):
|
||||
"""Executor for QwenImage models in ComfyUI."""
|
||||
class QwenImageAdapter(ComfyUIModelAdapter):
|
||||
model_types = ("qwen_image",)
|
||||
pipeline_class_name = "QwenImagePipeline"
|
||||
patch_size = 2
|
||||
|
||||
def __init__(self, generator, model_path, model, config):
|
||||
super().__init__(generator, model_path, model, config)
|
||||
self.patch_size = 2
|
||||
def pack(self, x, timestep, context, **kwargs) -> PackedForward:
|
||||
latents, orig_shape = self._pack_latents(x)
|
||||
seq = int(context.shape[-2]) if context.ndim >= 2 else int(context.shape[0])
|
||||
return PackedForward(
|
||||
latents=latents,
|
||||
timesteps=timestep * 1000.0,
|
||||
prompt_embeds=[context],
|
||||
prompt_seq_lens=[[seq]],
|
||||
height=orig_shape[-2] * 8,
|
||||
width=orig_shape[-1] * 8,
|
||||
unpack_ctx={
|
||||
"num_embeds": latents.shape[1],
|
||||
"orig_shape": orig_shape,
|
||||
"x": x,
|
||||
},
|
||||
)
|
||||
|
||||
def unpack(self, noise_pred, packed, x):
|
||||
ctx = packed.unpack_ctx
|
||||
return self._unpack_latents(
|
||||
noise_pred, ctx["num_embeds"], ctx["orig_shape"], ctx["x"]
|
||||
)
|
||||
|
||||
def _pack_latents(self, x):
|
||||
"""Process hidden states for QwenImage model."""
|
||||
latents = comfy.ldm.common_dit.pad_to_patch_size(
|
||||
x, (1, self.patch_size, self.patch_size)
|
||||
)
|
||||
@@ -47,8 +56,8 @@ class QwenImageExecutor(SGLDiffusionExecutor):
|
||||
)
|
||||
return latents, orig_shape
|
||||
|
||||
def _unpack_latents(self, latents, num_embeds, orig_shape, x):
|
||||
"""Unpack hidden states from packed format to standard format."""
|
||||
@staticmethod
|
||||
def _unpack_latents(latents, num_embeds, orig_shape, x):
|
||||
latents = latents[:, :num_embeds].view(
|
||||
orig_shape[0],
|
||||
orig_shape[-3],
|
||||
@@ -59,57 +68,14 @@ class QwenImageExecutor(SGLDiffusionExecutor):
|
||||
2,
|
||||
)
|
||||
latents = latents.permute(0, 4, 1, 2, 5, 3, 6)
|
||||
latents = latents.reshape(orig_shape)[:, :, :, : x.shape[-2], : x.shape[-1]]
|
||||
return latents
|
||||
|
||||
def forward(self, x, timestep, context, **kwargs):
|
||||
"""Forward pass for QwenImage model."""
|
||||
latents, orig_shape = self._pack_latents(x)
|
||||
num_embeds = latents.shape[1]
|
||||
height = orig_shape[-2] * 8
|
||||
width = orig_shape[-1] * 8
|
||||
|
||||
sampling_params = SamplingParams.from_user_sampling_params_args(
|
||||
self.model_path,
|
||||
server_args=self.generator.server_args,
|
||||
prompt=" ",
|
||||
guidance_scale=1.0,
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=1,
|
||||
num_inference_steps=1,
|
||||
save_output=False,
|
||||
suppress_logs=self.should_suppress_logs(timestep),
|
||||
)
|
||||
|
||||
# Prepare request (converts SamplingParams to Req)
|
||||
req = prepare_request(
|
||||
server_args=self.generator.server_args,
|
||||
sampling_params=sampling_params,
|
||||
)
|
||||
# Set ComfyUI-specific inputs directly on the Req object
|
||||
req.latents = latents
|
||||
req.timesteps = timestep * 1000.0
|
||||
req.prompt_embeds = [context]
|
||||
req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long)
|
||||
req.do_classifier_free_guidance = False
|
||||
req.generator = [
|
||||
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
|
||||
]
|
||||
|
||||
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
|
||||
noise_pred = output_batch.noise_pred
|
||||
|
||||
return self._unpack_latents(noise_pred, num_embeds, orig_shape, x)
|
||||
return latents.reshape(orig_shape)[:, :, :, : x.shape[-2], : x.shape[-1]]
|
||||
|
||||
|
||||
class QwenImageEditExecutor(QwenImageExecutor):
|
||||
"""Executor for QwenImageEdit models in ComfyUI."""
|
||||
class QwenImageEditAdapter(QwenImageAdapter):
|
||||
model_types = ("qwen_image_edit",)
|
||||
pipeline_class_name = "QwenImageEditPlusPipeline"
|
||||
|
||||
def __init__(self, generator, model_path, model, config):
|
||||
super().__init__(generator, model_path, model, config)
|
||||
|
||||
def forward(
|
||||
def pack(
|
||||
self,
|
||||
x,
|
||||
timestep,
|
||||
@@ -117,56 +83,22 @@ class QwenImageEditExecutor(QwenImageExecutor):
|
||||
attention_mask=None,
|
||||
ref_latents=None,
|
||||
additional_t_cond=None,
|
||||
transformer_options={},
|
||||
transformer_options=None,
|
||||
**kwargs,
|
||||
):
|
||||
"""Forward pass for QwenImageEdit model."""
|
||||
latents, orig_shape = self._pack_latents(x)
|
||||
num_embeds = latents.shape[1]
|
||||
height = orig_shape[-2] * 8
|
||||
width = orig_shape[-1] * 8
|
||||
) -> PackedForward:
|
||||
packed = super().pack(x, timestep, context, **kwargs)
|
||||
if ref_latents:
|
||||
pack_ref, orig_ref_shape = self._pack_latents(ref_latents[0])
|
||||
packed.extra_req["image_latent"] = pack_ref
|
||||
packed.extra_req["vae_image_sizes"] = [
|
||||
(orig_ref_shape[-1], orig_ref_shape[-2])
|
||||
]
|
||||
return packed
|
||||
|
||||
# Prepare vae_image_sizes for the condition image (ref_latents)
|
||||
vae_image_sizes = []
|
||||
pack_ref_latents = None
|
||||
|
||||
# TODO: sgld now don't support multiple condition images, so we only support one condition image for now.
|
||||
if ref_latents is not None and len(ref_latents) > 0:
|
||||
pack_ref_latents, orig_ref_shape = self._pack_latents(ref_latents[0])
|
||||
vae_image_sizes = [(orig_ref_shape[-1], orig_ref_shape[-2])]
|
||||
class QwenImageExecutor(SGLDiffusionExecutor):
|
||||
adapter_cls = QwenImageAdapter
|
||||
|
||||
sampling_params = SamplingParams.from_user_sampling_params_args(
|
||||
self.model_path,
|
||||
server_args=self.generator.server_args,
|
||||
prompt=" ",
|
||||
guidance_scale=1.0,
|
||||
image_path="",
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=1,
|
||||
num_inference_steps=1,
|
||||
save_output=False,
|
||||
suppress_logs=self.should_suppress_logs(timestep),
|
||||
)
|
||||
|
||||
# Prepare request (converts SamplingParams to Req)
|
||||
req = prepare_request(
|
||||
server_args=self.generator.server_args,
|
||||
sampling_params=sampling_params,
|
||||
)
|
||||
# Set ComfyUI-specific inputs directly on the Req object
|
||||
req.latents = latents
|
||||
req.image_latent = pack_ref_latents
|
||||
req.timesteps = timestep * 1000.0
|
||||
req.vae_image_sizes = vae_image_sizes
|
||||
req.prompt_embeds = [context]
|
||||
req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long)
|
||||
req.do_classifier_free_guidance = False
|
||||
req.generator = [
|
||||
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
|
||||
]
|
||||
|
||||
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
|
||||
noise_pred = output_batch.noise_pred
|
||||
|
||||
return self._unpack_latents(noise_pred, num_embeds, orig_shape, x)
|
||||
class QwenImageEditExecutor(SGLDiffusionExecutor):
|
||||
adapter_cls = QwenImageEditAdapter
|
||||
|
||||
@@ -1,64 +1,30 @@
|
||||
"""
|
||||
ZImage executor for SGLang Diffusion ComfyUI integration.
|
||||
"""
|
||||
|
||||
import torch
|
||||
|
||||
try:
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
|
||||
from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
|
||||
except ImportError:
|
||||
print(
|
||||
"Error: sglang.multimodal_gen is not installed. Please install it using 'pip install sglang[diffusion]'"
|
||||
)
|
||||
"""Z-Image (lumina2) adapter for the ComfyUI DiT-forward contract."""
|
||||
|
||||
from .adapter import ComfyUIModelAdapter, PackedForward
|
||||
from .base import SGLDiffusionExecutor
|
||||
|
||||
|
||||
class ZImageExecutor(SGLDiffusionExecutor):
|
||||
"""Executor for ZImage models in ComfyUI."""
|
||||
class ZImageAdapter(ComfyUIModelAdapter):
|
||||
model_types = ("lumina2",)
|
||||
pipeline_class_name = "ZImagePipeline"
|
||||
|
||||
def __init__(self, generator, model_path, model, config):
|
||||
super().__init__(generator, model_path, model, config)
|
||||
|
||||
def forward(self, x, timesteps, context, **kwargs):
|
||||
"""Forward pass for ZImage model."""
|
||||
B, C, H, W = x.shape
|
||||
height = H * 8
|
||||
width = W * 8
|
||||
sampling_params = SamplingParams.from_user_sampling_params_args(
|
||||
self.model_path,
|
||||
server_args=self.generator.server_args,
|
||||
prompt=" ",
|
||||
guidance_scale=1.0,
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=1, # For images
|
||||
num_inference_steps=1, # Single step for ComfyUI
|
||||
save_output=False,
|
||||
suppress_logs=self.should_suppress_logs(timesteps),
|
||||
)
|
||||
|
||||
# Prepare request (converts SamplingParams to Req)
|
||||
req = prepare_request(
|
||||
server_args=self.generator.server_args,
|
||||
sampling_params=sampling_params,
|
||||
)
|
||||
latents = x.unsqueeze(2)
|
||||
def pack(self, x, timestep, context, **kwargs) -> PackedForward:
|
||||
context = context.squeeze(0)
|
||||
# Set ComfyUI-specific inputs directly on the Req object
|
||||
req.latents = latents # ComfyUI's x parameter
|
||||
req.timesteps = timesteps * 1000.0 # ComfyUI's timesteps parameter
|
||||
req.prompt_embeds = [
|
||||
context
|
||||
] # ComfyUI's context parameter (must be List[Tensor])
|
||||
req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long)
|
||||
req.do_classifier_free_guidance = False
|
||||
req.generator = [
|
||||
torch.Generator("cuda") for _ in range(req.num_outputs_per_prompt)
|
||||
]
|
||||
return PackedForward(
|
||||
latents=x.unsqueeze(2),
|
||||
timesteps=timestep * 1000.0,
|
||||
prompt_embeds=[context],
|
||||
prompt_seq_lens=[[int(context.shape[0])]],
|
||||
height=x.shape[-2] * 8,
|
||||
width=x.shape[-1] * 8,
|
||||
)
|
||||
|
||||
output_batch = self.generator._send_to_scheduler_and_wait_for_response([req])
|
||||
noise_pred = output_batch.noise_pred
|
||||
def unpack(self, noise_pred, packed, x):
|
||||
# SGLD returns 5D [B, C, T, H, W]; ComfyUI samples 4D [B, C, H, W].
|
||||
if noise_pred.ndim == 5:
|
||||
noise_pred = noise_pred.squeeze(2)
|
||||
return noise_pred.to(x.device)
|
||||
|
||||
return noise_pred.permute(1, 0, 2, 3).to(x.device)
|
||||
|
||||
class ZImageExecutor(SGLDiffusionExecutor):
|
||||
adapter_cls = ZImageAdapter
|
||||
|
||||
@@ -10,6 +10,16 @@ import folder_paths
|
||||
import torch
|
||||
|
||||
from .core import SGLDiffusionGenerator, SGLDiffusionServerAPI
|
||||
|
||||
|
||||
def _enable_gguf_in_diffusion_models() -> None:
|
||||
"""Let SGLDUNETLoader list ``.gguf`` DiTs. Quantization lives in the file."""
|
||||
entry = folder_paths.folder_names_and_paths.get("diffusion_models")
|
||||
if entry is not None and len(entry) >= 2 and isinstance(entry[1], set):
|
||||
entry[1].add(".gguf")
|
||||
|
||||
|
||||
_enable_gguf_in_diffusion_models()
|
||||
from .utils import (
|
||||
convert_b64_to_tensor_image,
|
||||
convert_video_to_comfy_video,
|
||||
@@ -25,7 +35,14 @@ class SGLDOptions:
|
||||
"required": {},
|
||||
"optional": {
|
||||
"model_type": (
|
||||
["auto-detect", "qwen_image", "qwen_image_edit", "flux", "lumina2"],
|
||||
[
|
||||
"auto-detect",
|
||||
"qwen_image",
|
||||
"qwen_image_edit",
|
||||
"flux",
|
||||
"lumina2",
|
||||
"minimax_h3",
|
||||
],
|
||||
{"default": "auto-detect"},
|
||||
),
|
||||
"enable_torch_compile": (
|
||||
@@ -61,6 +78,25 @@ class SGLDOptions:
|
||||
"STRING",
|
||||
{"default": ""},
|
||||
),
|
||||
"dit_layerwise_offload": (
|
||||
"BOOLEAN",
|
||||
{"default": False},
|
||||
),
|
||||
"enable_cache_dit": (
|
||||
"BOOLEAN",
|
||||
{"default": False},
|
||||
),
|
||||
"quantization": (
|
||||
"STRING",
|
||||
{"default": ""},
|
||||
),
|
||||
"transformer_weights_path": (
|
||||
"STRING",
|
||||
{
|
||||
"default": "",
|
||||
"multiline": False,
|
||||
},
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -82,6 +118,10 @@ class SGLDOptions:
|
||||
dp_degree: int = 1,
|
||||
enable_cfg_parallel: bool = False,
|
||||
attention_backend: str = "",
|
||||
dit_layerwise_offload: bool = False,
|
||||
enable_cache_dit: bool = False,
|
||||
quantization: str = "",
|
||||
transformer_weights_path: str = "",
|
||||
):
|
||||
"""
|
||||
Build a dictionary of SGLang Diffusion runtime options.
|
||||
@@ -89,7 +129,12 @@ class SGLDOptions:
|
||||
# Convert -1 to None for optional parameters (matching ServerArgs defaults)
|
||||
ulysses_degree = None if ulysses_degree == -1 else ulysses_degree
|
||||
ring_degree = None if ring_degree == -1 else ring_degree
|
||||
tp_size = None if tp_size == -1 else tp_size
|
||||
sp_degree = None if sp_degree == -1 else sp_degree
|
||||
attention_backend = None if attention_backend == "" else attention_backend
|
||||
# dp_degree is a leftover alias; ServerArgs only has dp_size.
|
||||
if dp_degree not in (None, 1) and dp_size in (None, 1):
|
||||
dp_size = dp_degree
|
||||
|
||||
options = {
|
||||
"model_type": model_type,
|
||||
@@ -100,10 +145,20 @@ class SGLDOptions:
|
||||
"ulysses_degree": ulysses_degree,
|
||||
"ring_degree": ring_degree,
|
||||
"dp_size": dp_size,
|
||||
"dp_degree": dp_degree,
|
||||
"enable_cfg_parallel": enable_cfg_parallel,
|
||||
"attention_backend": attention_backend,
|
||||
"dit_layerwise_offload": dit_layerwise_offload,
|
||||
}
|
||||
if enable_cache_dit:
|
||||
options["enable_cache_dit"] = True
|
||||
quantization = (quantization or "").strip()
|
||||
if quantization:
|
||||
options["quantization"] = quantization
|
||||
transformer_weights_path = (transformer_weights_path or "").strip()
|
||||
if transformer_weights_path:
|
||||
# Same selector as `sglang serve --transformer-weights-path`:
|
||||
# local .gguf, owner/repo/path/file.gguf, or owner/repo:QUANT.
|
||||
options["transformer_weights_path"] = transformer_weights_path
|
||||
|
||||
# Strip None to keep payload clean
|
||||
options = {k: v for k, v in options.items() if v is not None}
|
||||
@@ -165,7 +220,7 @@ class SGLDLoraLoader:
|
||||
|
||||
class SGLDUNETLoader:
|
||||
def __init__(self):
|
||||
self.generator = SGLDiffusionGenerator()
|
||||
self.generator = SGLDiffusionGenerator.shared()
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
|
||||
@@ -1,13 +1,16 @@
|
||||
# ComfyUI SGLDiffusion Pipeline Tests
|
||||
|
||||
This directory contains tests for each ComfyUI pipeline integration.
|
||||
These e2e tests start the **native** Flux / Qwen-Image / Z-Image pipeline with
|
||||
`comfyui_mode=True` and feed pre-encoded latents, timesteps, and embeddings the
|
||||
way the plugin adapter would. Adapter, session, and CUDA IPC unit tests live
|
||||
under `python/sglang/multimodal_gen/test/unit/`.
|
||||
|
||||
## Test Files
|
||||
|
||||
- `test_zimage_pipeline.py` - Tests for ComfyUIZImagePipeline
|
||||
- `test_flux_pipeline.py` - Tests for ComfyUIFluxPipeline
|
||||
- `test_qwen_image_pipeline.py` - Tests for ComfyUIQwenImagePipeline
|
||||
- `test_qwen_image_edit_pipeline.py` - Tests for ComfyUIQwenImageEditPipeline (I2I/edit mode)
|
||||
- `test_zimage_pipeline.py` - Z-Image native pipeline under `--comfyui-mode`
|
||||
- `test_flux_pipeline.py` - Flux native pipeline under `--comfyui-mode`
|
||||
- `test_qwen_image_pipeline.py` - Qwen-Image native pipeline under `--comfyui-mode`
|
||||
- `test_qwen_image_edit_pipeline.py` - Qwen-Image-Edit native pipeline (I2I/edit; experimental)
|
||||
|
||||
## Running Tests
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Test for ComfyUIFluxPipeline with pass-through scheduler."""
|
||||
"""Test for FluxPipeline with pass-through scheduler."""
|
||||
|
||||
import os
|
||||
import sys
|
||||
@@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
|
||||
|
||||
|
||||
def test_comfyui_flux_pipeline_direct() -> None:
|
||||
"""Test ComfyUIFluxPipeline with custom inputs."""
|
||||
"""Test FluxPipeline with custom inputs."""
|
||||
model_path = os.environ.get(
|
||||
"SGLANG_TEST_FLUX_MODEL_PATH",
|
||||
"black-forest-labs/FLUX.1-dev", # Supports both safetensors file and diffusers format
|
||||
@@ -20,7 +20,8 @@ def test_comfyui_flux_pipeline_direct() -> None:
|
||||
|
||||
generator = DiffGenerator.from_pretrained(
|
||||
model_path=model_path,
|
||||
pipeline_class_name="ComfyUIFluxPipeline",
|
||||
model_id="FLUX.1-dev",
|
||||
pipeline_class_name="FluxPipeline",
|
||||
num_gpus=2,
|
||||
comfyui_mode=True,
|
||||
)
|
||||
@@ -68,6 +69,7 @@ def test_comfyui_flux_pipeline_direct() -> None:
|
||||
width=width,
|
||||
num_frames=1,
|
||||
num_inference_steps=1,
|
||||
guidance_scale=1.0,
|
||||
save_output=True,
|
||||
return_trajectory_latents=True,
|
||||
)
|
||||
@@ -83,6 +85,10 @@ def test_comfyui_flux_pipeline_direct() -> None:
|
||||
|
||||
clip_dim = 768
|
||||
req.prompt_embeds = [pooled_projections, encoder_hidden_states]
|
||||
req.prompt_seq_lens = [
|
||||
[int(pooled_projections.shape[0])],
|
||||
[encoder_seq_len],
|
||||
]
|
||||
|
||||
if req.guidance_scale > 1.0:
|
||||
dummy_neg_clip_embedding = torch.zeros(
|
||||
@@ -103,11 +109,15 @@ def test_comfyui_flux_pipeline_direct() -> None:
|
||||
dummy_neg_clip_embedding,
|
||||
negative_encoder_hidden_states,
|
||||
]
|
||||
req.negative_prompt_seq_lens = [
|
||||
[int(dummy_neg_clip_embedding.shape[0])],
|
||||
[encoder_seq_len],
|
||||
]
|
||||
else:
|
||||
req.negative_prompt_embeds = None
|
||||
|
||||
req.pooled_embeds = [pooled_projections]
|
||||
req.neg_pooled_embeds = []
|
||||
req.neg_pooled_embeds = [torch.zeros_like(pooled_projections)]
|
||||
|
||||
if (
|
||||
req.guidance_scale > 1.0
|
||||
|
||||
+3
-3
@@ -1,4 +1,4 @@
|
||||
"""Test for ComfyUIQwenImageEditPipeline with pass-through scheduler (I2I/edit mode)."""
|
||||
"""Test for QwenImageEditPlusPipeline with pass-through scheduler (I2I/edit mode)."""
|
||||
|
||||
import os
|
||||
import sys
|
||||
@@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
|
||||
|
||||
|
||||
def test_comfyui_qwen_image_edit_pipeline_direct() -> None:
|
||||
"""Test ComfyUIQwenImageEditPipeline with edit mode (I2I) and custom inputs."""
|
||||
"""Test QwenImageEditPlusPipeline with edit mode (I2I) and custom inputs."""
|
||||
model_path = os.environ.get(
|
||||
"SGLANG_TEST_QWEN_IMAGE_EDIT_MODEL_PATH",
|
||||
"Qwen/Qwen-Image-Edit-2511", # Supports both safetensors file and diffusers format
|
||||
@@ -20,7 +20,7 @@ def test_comfyui_qwen_image_edit_pipeline_direct() -> None:
|
||||
|
||||
generator = DiffGenerator.from_pretrained(
|
||||
model_path=model_path,
|
||||
pipeline_class_name="ComfyUIQwenImageEditPipeline",
|
||||
pipeline_class_name="QwenImageEditPlusPipeline",
|
||||
num_gpus=1,
|
||||
comfyui_mode=True,
|
||||
dit_layerwise_offload=False,
|
||||
|
||||
+3
-3
@@ -1,4 +1,4 @@
|
||||
"""Test for ComfyUIQwenImagePipeline with pass-through scheduler."""
|
||||
"""Test for QwenImagePipeline with pass-through scheduler."""
|
||||
|
||||
import os
|
||||
import sys
|
||||
@@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
|
||||
|
||||
|
||||
def test_comfyui_qwen_image_pipeline_direct() -> None:
|
||||
"""Test ComfyUIQwenImagePipeline with custom inputs."""
|
||||
"""Test QwenImagePipeline with custom inputs."""
|
||||
model_path = os.environ.get(
|
||||
"SGLANG_TEST_QWEN_IMAGE_MODEL_PATH",
|
||||
"Qwen/Qwen-Image", # Supports both safetensors file and diffusers format
|
||||
@@ -20,7 +20,7 @@ def test_comfyui_qwen_image_pipeline_direct() -> None:
|
||||
|
||||
generator = DiffGenerator.from_pretrained(
|
||||
model_path=model_path,
|
||||
pipeline_class_name="ComfyUIQwenImagePipeline",
|
||||
pipeline_class_name="QwenImagePipeline",
|
||||
num_gpus=2,
|
||||
comfyui_mode=True,
|
||||
dit_layerwise_offload=False,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Test for ComfyUIZImagePipeline with pass-through scheduler."""
|
||||
"""Test for ZImagePipeline with pass-through scheduler."""
|
||||
|
||||
import os
|
||||
import sys
|
||||
@@ -12,7 +12,7 @@ from sglang.multimodal_gen.runtime.entrypoints.utils import prepare_request
|
||||
|
||||
|
||||
def test_comfyui_zimage_pipeline_direct() -> None:
|
||||
"""Test ComfyUIZImagePipeline with custom inputs."""
|
||||
"""Test ZImagePipeline with custom inputs."""
|
||||
model_path = os.environ.get(
|
||||
"SGLANG_TEST_ZIMAGE_MODEL_PATH",
|
||||
"Tongyi-MAI/Z-Image-Turbo", # Supports both safetensors file and diffusers format
|
||||
@@ -20,7 +20,7 @@ def test_comfyui_zimage_pipeline_direct() -> None:
|
||||
|
||||
generator = DiffGenerator.from_pretrained(
|
||||
model_path=model_path,
|
||||
pipeline_class_name="ComfyUIZImagePipeline",
|
||||
pipeline_class_name="ZImagePipeline",
|
||||
num_gpus=1,
|
||||
sp_degree=1,
|
||||
comfyui_mode=True,
|
||||
@@ -77,6 +77,7 @@ def test_comfyui_zimage_pipeline_direct() -> None:
|
||||
req.latents = latents
|
||||
req.timesteps = timesteps
|
||||
req.prompt_embeds = [context]
|
||||
req.prompt_seq_lens = [[context_seq_len]]
|
||||
req.negative_prompt_embeds = None
|
||||
req.raw_latent_shape = torch.tensor(latents.shape, dtype=torch.long)
|
||||
|
||||
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
{
|
||||
"6": {
|
||||
"class_type": "SGLDUNETLoader",
|
||||
"inputs": {
|
||||
"unet_name": "minimax_h3_ref2va_bf16.safetensors",
|
||||
"weight_dtype": "default",
|
||||
"sgld_options": ["52", 0]
|
||||
}
|
||||
},
|
||||
"52": {
|
||||
"class_type": "SGLDOptions",
|
||||
"inputs": {
|
||||
"model_type": "minimax_h3",
|
||||
"enable_torch_compile": false,
|
||||
"num_gpus": 1,
|
||||
"tp_size": -1,
|
||||
"sp_degree": -1,
|
||||
"ulysses_degree": -1,
|
||||
"ring_degree": -1,
|
||||
"dp_size": 1,
|
||||
"dp_degree": 1,
|
||||
"enable_cfg_parallel": false,
|
||||
"attention_backend": ""
|
||||
}
|
||||
},
|
||||
"13": {
|
||||
"class_type": "CLIPLoader",
|
||||
"inputs": {
|
||||
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
|
||||
"type": "minimax",
|
||||
"device": "default"
|
||||
}
|
||||
},
|
||||
"11": {
|
||||
"class_type": "VAELoader",
|
||||
"inputs": {
|
||||
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
|
||||
}
|
||||
},
|
||||
"24": {
|
||||
"class_type": "VAELoader",
|
||||
"inputs": {
|
||||
"vae_name": "minimax_h3_audio_vae_fp32.safetensors"
|
||||
}
|
||||
},
|
||||
"137": {
|
||||
"class_type": "LoadImage",
|
||||
"inputs": {
|
||||
"image": "sglang_girl.jpg"
|
||||
}
|
||||
},
|
||||
"136": {
|
||||
"class_type": "MiniMaxH3ReferenceToVideo",
|
||||
"inputs": {
|
||||
"clip": ["13", 0],
|
||||
"vae": ["11", 0],
|
||||
"audio_vae": ["24", 0],
|
||||
"prompt": "Use <Picture 1> as the sole character and style reference. A young woman walks slowly through a quiet garden at golden hour, looking toward the camera, then turning to watch leaves drift past. Soft cinematic lighting, shallow depth of field, gentle breeze in her hair.\n\nAudio: quiet outdoor ambience, faint birds, a soft piano motif, no lyrics.",
|
||||
"width": 864,
|
||||
"height": 480,
|
||||
"length": 73,
|
||||
"ref_image_size": "match",
|
||||
"ref_images.ref_image_0": ["137", 0]
|
||||
}
|
||||
},
|
||||
"17": {
|
||||
"class_type": "KSamplerSelect",
|
||||
"inputs": {
|
||||
"sampler_name": "res_multistep"
|
||||
}
|
||||
},
|
||||
"9": {
|
||||
"class_type": "BasicScheduler",
|
||||
"inputs": {
|
||||
"model": ["6", 0],
|
||||
"scheduler": "beta",
|
||||
"steps": 8,
|
||||
"denoise": 1.0
|
||||
}
|
||||
},
|
||||
"15": {
|
||||
"class_type": "RandomNoise",
|
||||
"inputs": {
|
||||
"noise_seed": 1101
|
||||
}
|
||||
},
|
||||
"16": {
|
||||
"class_type": "BasicGuider",
|
||||
"inputs": {
|
||||
"model": ["6", 0],
|
||||
"conditioning": ["136", 0]
|
||||
}
|
||||
},
|
||||
"14": {
|
||||
"class_type": "SamplerCustomAdvanced",
|
||||
"inputs": {
|
||||
"noise": ["15", 0],
|
||||
"guider": ["16", 0],
|
||||
"sampler": ["17", 0],
|
||||
"sigmas": ["9", 0],
|
||||
"latent_image": ["136", 1]
|
||||
}
|
||||
},
|
||||
"10": {
|
||||
"class_type": "VAEDecode",
|
||||
"inputs": {
|
||||
"samples": ["14", 0],
|
||||
"vae": ["11", 0]
|
||||
}
|
||||
},
|
||||
"23": {
|
||||
"class_type": "VAEDecodeAudio",
|
||||
"inputs": {
|
||||
"samples": ["14", 0],
|
||||
"vae": ["24", 0]
|
||||
}
|
||||
},
|
||||
"91": {
|
||||
"class_type": "CreateVideo",
|
||||
"inputs": {
|
||||
"images": ["10", 0],
|
||||
"audio": ["23", 0],
|
||||
"fps": 24,
|
||||
"bit_depth": 8
|
||||
}
|
||||
},
|
||||
"92": {
|
||||
"class_type": "SaveVideo",
|
||||
"inputs": {
|
||||
"video": ["91", 0],
|
||||
"filename_prefix": "video/MiniMax_H3_sgld_r2v",
|
||||
"format": "auto",
|
||||
"codec": "auto"
|
||||
}
|
||||
}
|
||||
}
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
{
|
||||
"6": {
|
||||
"class_type": "SGLDUNETLoader",
|
||||
"inputs": {
|
||||
"unet_name": "minimax_h3_fl2va_bf16.safetensors",
|
||||
"weight_dtype": "default",
|
||||
"sgld_options": ["52", 0]
|
||||
},
|
||||
"_meta": {
|
||||
"title": "SGLDiffusion UNET Loader"
|
||||
}
|
||||
},
|
||||
"52": {
|
||||
"class_type": "SGLDOptions",
|
||||
"inputs": {
|
||||
"model_type": "minimax_h3",
|
||||
"enable_torch_compile": false,
|
||||
"num_gpus": 1,
|
||||
"tp_size": -1,
|
||||
"sp_degree": -1,
|
||||
"ulysses_degree": -1,
|
||||
"ring_degree": -1,
|
||||
"dp_size": 1,
|
||||
"dp_degree": 1,
|
||||
"enable_cfg_parallel": false,
|
||||
"attention_backend": ""
|
||||
},
|
||||
"_meta": {
|
||||
"title": "SGLDiffusion Options"
|
||||
}
|
||||
},
|
||||
"13": {
|
||||
"class_type": "CLIPLoader",
|
||||
"inputs": {
|
||||
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
|
||||
"type": "minimax",
|
||||
"device": "default"
|
||||
}
|
||||
},
|
||||
"11": {
|
||||
"class_type": "VAELoader",
|
||||
"inputs": {
|
||||
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
|
||||
}
|
||||
},
|
||||
"24": {
|
||||
"class_type": "VAELoader",
|
||||
"inputs": {
|
||||
"vae_name": "minimax_h3_audio_vae_fp32.safetensors"
|
||||
}
|
||||
},
|
||||
"104": {
|
||||
"class_type": "MiniMaxH3ImageToVideo",
|
||||
"inputs": {
|
||||
"clip": ["13", 0],
|
||||
"vae": ["11", 0],
|
||||
"prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.",
|
||||
"width": 864,
|
||||
"height": 480,
|
||||
"length": 73
|
||||
}
|
||||
},
|
||||
"17": {
|
||||
"class_type": "KSamplerSelect",
|
||||
"inputs": {
|
||||
"sampler_name": "res_multistep"
|
||||
}
|
||||
},
|
||||
"9": {
|
||||
"class_type": "BasicScheduler",
|
||||
"inputs": {
|
||||
"model": ["6", 0],
|
||||
"scheduler": "simple",
|
||||
"steps": 8,
|
||||
"denoise": 1.0
|
||||
}
|
||||
},
|
||||
"15": {
|
||||
"class_type": "RandomNoise",
|
||||
"inputs": {
|
||||
"noise_seed": 1101
|
||||
}
|
||||
},
|
||||
"16": {
|
||||
"class_type": "BasicGuider",
|
||||
"inputs": {
|
||||
"model": ["6", 0],
|
||||
"conditioning": ["104", 0]
|
||||
}
|
||||
},
|
||||
"14": {
|
||||
"class_type": "SamplerCustomAdvanced",
|
||||
"inputs": {
|
||||
"noise": ["15", 0],
|
||||
"guider": ["16", 0],
|
||||
"sampler": ["17", 0],
|
||||
"sigmas": ["9", 0],
|
||||
"latent_image": ["104", 1]
|
||||
}
|
||||
},
|
||||
"10": {
|
||||
"class_type": "VAEDecode",
|
||||
"inputs": {
|
||||
"samples": ["14", 0],
|
||||
"vae": ["11", 0]
|
||||
}
|
||||
},
|
||||
"23": {
|
||||
"class_type": "VAEDecodeAudio",
|
||||
"inputs": {
|
||||
"samples": ["14", 0],
|
||||
"vae": ["24", 0]
|
||||
}
|
||||
},
|
||||
"91": {
|
||||
"class_type": "CreateVideo",
|
||||
"inputs": {
|
||||
"images": ["10", 0],
|
||||
"audio": ["23", 0],
|
||||
"fps": 24,
|
||||
"bit_depth": 8
|
||||
}
|
||||
},
|
||||
"92": {
|
||||
"class_type": "SaveVideo",
|
||||
"inputs": {
|
||||
"video": ["91", 0],
|
||||
"filename_prefix": "video/MiniMax_H3_sgld_t2v",
|
||||
"format": "auto",
|
||||
"codec": "auto"
|
||||
}
|
||||
}
|
||||
}
|
||||
+241
@@ -0,0 +1,241 @@
|
||||
{
|
||||
"6": {
|
||||
"class_type": "SGLDUNETLoader",
|
||||
"inputs": {
|
||||
"unet_name": "minimax_h3_fl2va_bf16.safetensors",
|
||||
"weight_dtype": "default",
|
||||
"sgld_options": ["52", 0]
|
||||
}
|
||||
},
|
||||
"7": {
|
||||
"class_type": "SGLDLoraLoader",
|
||||
"inputs": {
|
||||
"model": ["6", 0],
|
||||
"lora_name": "minimax_h3_fl2v_lightx2v_turbo_4step_v0.1_comfy.safetensors",
|
||||
"strength_model": 1.0,
|
||||
"nickname": "lightx2v",
|
||||
"target": "all"
|
||||
}
|
||||
},
|
||||
"52": {
|
||||
"class_type": "SGLDOptions",
|
||||
"inputs": {
|
||||
"model_type": "minimax_h3",
|
||||
"enable_torch_compile": false,
|
||||
"num_gpus": 1,
|
||||
"tp_size": -1,
|
||||
"sp_degree": -1,
|
||||
"ulysses_degree": -1,
|
||||
"ring_degree": -1,
|
||||
"dp_size": 1,
|
||||
"dp_degree": 1,
|
||||
"enable_cfg_parallel": false,
|
||||
"attention_backend": "",
|
||||
"dit_layerwise_offload": false,
|
||||
"enable_cache_dit": false,
|
||||
"quantization": "",
|
||||
"transformer_weights_path": ""
|
||||
}
|
||||
},
|
||||
"13": {
|
||||
"class_type": "CLIPLoader",
|
||||
"inputs": {
|
||||
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
|
||||
"type": "minimax",
|
||||
"device": "default"
|
||||
}
|
||||
},
|
||||
"11": {
|
||||
"class_type": "VAELoader",
|
||||
"inputs": {
|
||||
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
|
||||
}
|
||||
},
|
||||
"24": {
|
||||
"class_type": "VAELoader",
|
||||
"inputs": {
|
||||
"vae_name": "minimax_h3_audio_vae_fp32.safetensors"
|
||||
}
|
||||
},
|
||||
"104": {
|
||||
"class_type": "MiniMaxH3ImageToVideo",
|
||||
"inputs": {
|
||||
"clip": ["13", 0],
|
||||
"vae": ["11", 0],
|
||||
"prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.",
|
||||
"width": 768,
|
||||
"height": 448,
|
||||
"length": 39
|
||||
}
|
||||
},
|
||||
"17": {
|
||||
"class_type": "KSamplerSelect",
|
||||
"inputs": {
|
||||
"sampler_name": "res_multistep"
|
||||
}
|
||||
},
|
||||
"9": {
|
||||
"class_type": "BasicScheduler",
|
||||
"inputs": {
|
||||
"model": ["7", 0],
|
||||
"scheduler": "simple",
|
||||
"steps": 4,
|
||||
"denoise": 1.0
|
||||
}
|
||||
},
|
||||
"15": {
|
||||
"class_type": "RandomNoise",
|
||||
"inputs": {
|
||||
"noise_seed": 7703
|
||||
}
|
||||
},
|
||||
"16": {
|
||||
"class_type": "BasicGuider",
|
||||
"inputs": {
|
||||
"model": ["7", 0],
|
||||
"conditioning": ["104", 0]
|
||||
}
|
||||
},
|
||||
"14": {
|
||||
"class_type": "SamplerCustomAdvanced",
|
||||
"inputs": {
|
||||
"noise": ["15", 0],
|
||||
"guider": ["16", 0],
|
||||
"sampler": ["17", 0],
|
||||
"sigmas": ["9", 0],
|
||||
"latent_image": ["104", 1]
|
||||
}
|
||||
},
|
||||
"93": {
|
||||
"class_type": "VAEDecode",
|
||||
"inputs": {
|
||||
"samples": ["14", 0],
|
||||
"vae": ["11", 0]
|
||||
}
|
||||
},
|
||||
"23": {
|
||||
"class_type": "VAEDecodeAudio",
|
||||
"inputs": {
|
||||
"samples": ["14", 0],
|
||||
"vae": ["24", 0]
|
||||
}
|
||||
},
|
||||
"94": {
|
||||
"class_type": "CreateVideo",
|
||||
"inputs": {
|
||||
"images": ["93", 0],
|
||||
"audio": ["23", 0],
|
||||
"fps": 24,
|
||||
"bit_depth": 8
|
||||
}
|
||||
},
|
||||
"95": {
|
||||
"class_type": "SaveVideo",
|
||||
"inputs": {
|
||||
"video": ["94", 0],
|
||||
"filename_prefix": "video/MiniMax_H3_sgld_t2v_upscaler_low",
|
||||
"format": "auto",
|
||||
"codec": "auto"
|
||||
}
|
||||
},
|
||||
"201": {
|
||||
"class_type": "LTXVSeparateAVLatent",
|
||||
"inputs": {
|
||||
"av_latent": ["14", 0]
|
||||
}
|
||||
},
|
||||
"202": {
|
||||
"class_type": "MinimaxH3LatentUpscaler3D",
|
||||
"inputs": {
|
||||
"latent": ["201", 0],
|
||||
"model_name": "minimax_h3_latent_upscaler_3d_fp16.safetensors",
|
||||
"mode": "scale by multiplier",
|
||||
"mode.scale": 2.0,
|
||||
"align": 32,
|
||||
"device": "cuda",
|
||||
"precision": "fp16"
|
||||
}
|
||||
},
|
||||
"203": {
|
||||
"class_type": "LTXVConcatAVLatent",
|
||||
"inputs": {
|
||||
"video_latent": ["202", 0],
|
||||
"audio_latent": ["201", 1]
|
||||
}
|
||||
},
|
||||
"204": {
|
||||
"class_type": "MiniMaxH3ImageToVideo",
|
||||
"inputs": {
|
||||
"clip": ["13", 0],
|
||||
"vae": ["11", 0],
|
||||
"prompt": "Vaporwave title sequence look: pink and blue gradient palette, VHS tracking artifacts, Greek statue motifs, chrome palm trees, RGB chromatic aberration, lo-fi retro atmosphere, mood languid and nostalgic.\n\nTimeline:\n[0s-1s] VHS static opens the frame, the title \"COMFYUI\" appears with RGB split and a slight horizontal jitter.\n[1s-2.5s] Hard cut, a Greek plaster bust close-up, pink-purple gradient sky, a pixelated sun.\n[2.5s-4s] Clean \"STARRING\" credits appear, \"LATENT\" and \"CONTROLNET\" each shown exactly once.\n[4s-5s] Final card \"DIRECTED BY COMFYUI\" holds, one VHS tracking glitch settling into stability.\n\nHard cuts only, transitions landing with tape jumps, no push-ins, no dissolves.\n\nAudio: lo-fi vaporwave score, slow drum machine with soft bass, VHS tape-noise sample joins at 2.5s, melody fading for the last 1s.\n\nAll text must be clearly legible, do not misspell English, no Chinese characters, do not repeat names or job titles, no soft dissolves, no subtitle bars.",
|
||||
"width": 1536,
|
||||
"height": 896,
|
||||
"length": 39
|
||||
}
|
||||
},
|
||||
"209": {
|
||||
"class_type": "BasicScheduler",
|
||||
"inputs": {
|
||||
"model": ["7", 0],
|
||||
"scheduler": "simple",
|
||||
"steps": 4,
|
||||
"denoise": 0.45
|
||||
}
|
||||
},
|
||||
"215": {
|
||||
"class_type": "RandomNoise",
|
||||
"inputs": {
|
||||
"noise_seed": 7704
|
||||
}
|
||||
},
|
||||
"216": {
|
||||
"class_type": "BasicGuider",
|
||||
"inputs": {
|
||||
"model": ["7", 0],
|
||||
"conditioning": ["204", 0]
|
||||
}
|
||||
},
|
||||
"214": {
|
||||
"class_type": "SamplerCustomAdvanced",
|
||||
"inputs": {
|
||||
"noise": ["215", 0],
|
||||
"guider": ["216", 0],
|
||||
"sampler": ["17", 0],
|
||||
"sigmas": ["209", 0],
|
||||
"latent_image": ["203", 0]
|
||||
}
|
||||
},
|
||||
"10": {
|
||||
"class_type": "VAEDecode",
|
||||
"inputs": {
|
||||
"samples": ["214", 0],
|
||||
"vae": ["11", 0]
|
||||
}
|
||||
},
|
||||
"96": {
|
||||
"class_type": "VAEDecodeAudio",
|
||||
"inputs": {
|
||||
"samples": ["214", 0],
|
||||
"vae": ["24", 0]
|
||||
}
|
||||
},
|
||||
"91": {
|
||||
"class_type": "CreateVideo",
|
||||
"inputs": {
|
||||
"images": ["10", 0],
|
||||
"audio": ["96", 0],
|
||||
"fps": 24,
|
||||
"bit_depth": 8
|
||||
}
|
||||
},
|
||||
"92": {
|
||||
"class_type": "SaveVideo",
|
||||
"inputs": {
|
||||
"video": ["91", 0],
|
||||
"filename_prefix": "video/MiniMax_H3_sgld_t2v_upscaler",
|
||||
"format": "auto",
|
||||
"codec": "auto"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -104,6 +104,7 @@ class MiniMaxH3DiTArchConfig(DiTArchConfig):
|
||||
norm_eps: float = 1e-5
|
||||
qk_norm_eps: float = 1e-5
|
||||
final_norm_eps: float = 1e-5
|
||||
qkv_checkpoint_grouped: bool = True
|
||||
checkpoint_uses_diffusers_layout: bool = False
|
||||
adaln_affine_input_dim: int | None = None
|
||||
has_gate_compress: bool = False
|
||||
|
||||
@@ -33,6 +33,11 @@ class MiniMaxH3AudioVAEConfig(VAEConfig):
|
||||
super().update_model_arch(model_dict)
|
||||
|
||||
def post_init(self) -> None:
|
||||
if (
|
||||
self.arch_config.latents_mean is None
|
||||
and self.arch_config.latents_std is None
|
||||
):
|
||||
return
|
||||
validate_minimax_h3_vae_latent_stats(
|
||||
self.arch_config,
|
||||
component_name="audio_vae",
|
||||
|
||||
@@ -69,6 +69,12 @@ class MiniMaxH3VideoVAEConfig(VAEConfig):
|
||||
|
||||
def post_init(self) -> None:
|
||||
self.resolved_parallel_decode_mode()
|
||||
if (
|
||||
self.arch_config.latents_mean is None
|
||||
and self.arch_config.latents_std is None
|
||||
):
|
||||
# ComfyUI DiT-forward never loads this VAE; stats stay unset.
|
||||
return
|
||||
validate_minimax_h3_vae_latent_stats(
|
||||
self.arch_config,
|
||||
component_name="video_vae",
|
||||
|
||||
@@ -54,6 +54,10 @@ from sglang.multimodal_gen.runtime.disaggregation.transport.protocol import (
|
||||
encode_transfer_msg,
|
||||
is_transfer_message,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.distributed.ipc_cuda import (
|
||||
attach_cuda_tensors,
|
||||
detach_cuda_tensors,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.distributed.utils import broadcast_pyobj
|
||||
from sglang.multimodal_gen.runtime.entrypoints.utils import expand_request_outputs
|
||||
from sglang.multimodal_gen.runtime.pipelines_core import Req
|
||||
@@ -793,6 +797,31 @@ class SchedulerDisaggMixin:
|
||||
|
||||
return data
|
||||
|
||||
def _broadcast_recv_reqs(self: Scheduler, recv_reqs):
|
||||
"""ComfyUI multi-rank recv: pickle the Req skeleton, NCCL the CUDA tensors.
|
||||
|
||||
The general SP/CFG/TP path stays in ``Scheduler.recv_reqs`` as the
|
||||
original whole-list ``broadcast_pyobj``. This helper is only the
|
||||
ComfyUI overlay and does not use disagg extract.
|
||||
"""
|
||||
is_rank0 = self.gpu_id == 0
|
||||
if is_rank0:
|
||||
assert recv_reqs is not None, "rank 0 must pass the ZMQ poll result"
|
||||
skeleton, tensors = detach_cuda_tensors(recv_reqs)
|
||||
else:
|
||||
skeleton, tensors = None, None
|
||||
|
||||
skeleton = self._broadcast_to_all_ranks(skeleton)
|
||||
tensors = self._broadcast_tensor_dict_to_all_ranks(tensors)
|
||||
if is_rank0:
|
||||
return recv_reqs
|
||||
if not skeleton:
|
||||
return []
|
||||
local_device = torch.device(
|
||||
f"{current_platform.device_type}:{self.worker.local_rank}"
|
||||
)
|
||||
return attach_cuda_tensors(skeleton, tensors or {}, device=local_device)
|
||||
|
||||
def _is_multi_rank(self: Scheduler) -> bool:
|
||||
sa = self.server_args
|
||||
return sa.sp_degree != 1 or sa.tp_size > 1 or sa.enable_cfg_parallel
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Keep CUDA tensors off the pickle path for local scheduler hops.
|
||||
|
||||
Two hops share the same tree walk:
|
||||
|
||||
- ComfyUI / DiffGenerator ↔ rank-0: replace tensors with ``CudaIpcRef``
|
||||
handles before pickle, rebuild with ``UntypedStorage._new_shared_cuda``
|
||||
- ComfyUI multi-rank recv: detach tensors for NCCL, pickle the skeleton,
|
||||
attach them again (not the disagg extract path)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import dataclasses
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
# Producer-side retain: CUDA IPC cannot open a handle in the same process,
|
||||
# and the allocation must stay alive until the consumer maps it.
|
||||
# Do not LRU-evict: a live handle that is dropped before the peer maps it
|
||||
# silently corrupts the hop.
|
||||
_PRODUCER_TENSORS: OrderedDict[tuple[bytes, int], torch.Tensor] = OrderedDict()
|
||||
_WARNED_ASYNC_ALLOC = False
|
||||
|
||||
|
||||
def _producer_key(handle: bytes, storage_offset_bytes: int) -> tuple[bytes, int]:
|
||||
return (handle, storage_offset_bytes)
|
||||
|
||||
|
||||
def _retain_producer_tensor(
|
||||
handle: bytes, storage_offset_bytes: int, tensor: torch.Tensor
|
||||
) -> None:
|
||||
key = _producer_key(handle, storage_offset_bytes)
|
||||
_PRODUCER_TENSORS[key] = tensor
|
||||
_PRODUCER_TENSORS.move_to_end(key)
|
||||
|
||||
|
||||
def release_retained_producer_tensors() -> None:
|
||||
"""Drop producer-side IPC retains after the peer has mapped them."""
|
||||
_PRODUCER_TENSORS.clear()
|
||||
|
||||
|
||||
@dataclass
|
||||
class CudaIpcRef:
|
||||
"""Picklable CUDA IPC handle plus enough metadata to rebuild the tensor."""
|
||||
|
||||
dtype: str
|
||||
shape: tuple[int, ...]
|
||||
stride: tuple[int, ...]
|
||||
storage_offset: int
|
||||
device_index: int
|
||||
handle: bytes
|
||||
storage_size_bytes: int
|
||||
storage_offset_bytes: int
|
||||
ref_counter_handle: bytes
|
||||
ref_counter_offset: int
|
||||
event_handle: bytes
|
||||
event_sync_required: bool
|
||||
|
||||
@classmethod
|
||||
def from_tensor(cls, tensor: torch.Tensor) -> CudaIpcRef:
|
||||
if not tensor.is_cuda:
|
||||
raise TypeError("CudaIpcRef only shares CUDA tensors")
|
||||
tensor = tensor.detach().contiguous()
|
||||
(
|
||||
device_index,
|
||||
handle,
|
||||
storage_size_bytes,
|
||||
storage_offset_bytes,
|
||||
ref_counter_handle,
|
||||
ref_counter_offset,
|
||||
event_handle,
|
||||
event_sync_required,
|
||||
) = tensor.untyped_storage()._share_cuda_()
|
||||
ref = cls(
|
||||
dtype=str(tensor.dtype).removeprefix("torch."),
|
||||
shape=tuple(tensor.shape),
|
||||
stride=tuple(tensor.stride()),
|
||||
storage_offset=int(tensor.storage_offset()),
|
||||
device_index=int(device_index),
|
||||
handle=handle,
|
||||
storage_size_bytes=int(storage_size_bytes),
|
||||
storage_offset_bytes=int(storage_offset_bytes),
|
||||
ref_counter_handle=ref_counter_handle,
|
||||
ref_counter_offset=int(ref_counter_offset),
|
||||
event_handle=event_handle,
|
||||
event_sync_required=bool(event_sync_required),
|
||||
)
|
||||
_retain_producer_tensor(handle, int(storage_offset_bytes), tensor)
|
||||
return ref
|
||||
|
||||
def materialize(self) -> torch.Tensor:
|
||||
dtype = getattr(torch, self.dtype)
|
||||
if self.handle is None or self.storage_size_bytes == 0:
|
||||
return torch.empty(
|
||||
self.shape, dtype=dtype, device=f"cuda:{self.device_index}"
|
||||
)
|
||||
|
||||
local = _PRODUCER_TENSORS.pop(
|
||||
_producer_key(self.handle, self.storage_offset_bytes), None
|
||||
)
|
||||
if local is not None:
|
||||
return local.detach().clone()
|
||||
|
||||
torch.cuda._lazy_init()
|
||||
storage = torch.UntypedStorage._new_shared_cuda(
|
||||
self.device_index,
|
||||
self.handle,
|
||||
self.storage_size_bytes,
|
||||
self.storage_offset_bytes,
|
||||
self.ref_counter_handle,
|
||||
self.ref_counter_offset,
|
||||
self.event_handle,
|
||||
self.event_sync_required,
|
||||
)
|
||||
typed = torch.storage.TypedStorage(
|
||||
wrap_storage=storage, dtype=dtype, _internal=True
|
||||
)
|
||||
mapped = torch._utils._rebuild_tensor(
|
||||
typed, self.storage_offset, self.shape, self.stride
|
||||
)
|
||||
# Own a private copy so the IPC mapping can be released immediately.
|
||||
return mapped.clone()
|
||||
|
||||
|
||||
def spill_cuda_tensors(value: Any, *, in_place: bool = False) -> Any:
|
||||
"""Replace CUDA tensors with ``CudaIpcRef`` handles.
|
||||
|
||||
By default dataclasses are shallow-copied so the caller's tensors stay
|
||||
intact (needed on the client, which still holds the original ``Req``).
|
||||
Scheduler replies can pass ``in_place=True``.
|
||||
"""
|
||||
return _map_tree(value, _spill_one, copy_dataclasses=not in_place)
|
||||
|
||||
|
||||
def materialize_cuda_refs(value: Any) -> Any:
|
||||
"""Rebuild CUDA tensors from ``CudaIpcRef`` handles, in place on dataclasses."""
|
||||
return _map_tree(value, _materialize_one, copy_dataclasses=False)
|
||||
|
||||
|
||||
def _spill_one(value: Any) -> Any:
|
||||
if isinstance(value, torch.Tensor) and value.is_cuda:
|
||||
try:
|
||||
return CudaIpcRef.from_tensor(value)
|
||||
except RuntimeError as exc:
|
||||
# cudaMallocAsync (ComfyUI default) cannot export IPC handles.
|
||||
# Leave the tensor in place so pickle falls back to a host copy.
|
||||
msg = str(exc)
|
||||
if "shareIpcHandle" not in msg and "cudaMallocAsync" not in msg:
|
||||
raise
|
||||
global _WARNED_ASYNC_ALLOC
|
||||
if not _WARNED_ASYNC_ALLOC:
|
||||
_WARNED_ASYNC_ALLOC = True
|
||||
logger.warning(
|
||||
"CUDA IPC export failed (%s); falling back to a host pickle "
|
||||
"copy. ComfyUI's default cudaMallocAsync pool cannot export "
|
||||
"IPC handles.",
|
||||
exc,
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _materialize_one(value: Any) -> Any:
|
||||
if isinstance(value, CudaIpcRef):
|
||||
return value.materialize()
|
||||
return value
|
||||
|
||||
|
||||
def _map_tree(value: Any, fn, copy_dataclasses: bool) -> Any:
|
||||
replaced = fn(value)
|
||||
if replaced is not value:
|
||||
return replaced
|
||||
if isinstance(value, list):
|
||||
return [_map_tree(item, fn, copy_dataclasses) for item in value]
|
||||
if isinstance(value, tuple):
|
||||
return tuple(_map_tree(item, fn, copy_dataclasses) for item in value)
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
key: _map_tree(item, fn, copy_dataclasses) for key, item in value.items()
|
||||
}
|
||||
if dataclasses.is_dataclass(value) and not isinstance(value, type):
|
||||
if copy_dataclasses:
|
||||
value = copy.copy(value)
|
||||
for field in dataclasses.fields(value):
|
||||
current = getattr(value, field.name, None)
|
||||
updated = _map_tree(current, fn, copy_dataclasses)
|
||||
if updated is not current:
|
||||
setattr(value, field.name, updated)
|
||||
return value
|
||||
return value
|
||||
|
||||
|
||||
_SEP = "\x1f"
|
||||
|
||||
|
||||
def detach_cuda_tensors(value: Any) -> tuple[Any, dict[str, torch.Tensor]]:
|
||||
"""Copy the tree, replace CUDA tensors with ``None``, collect them by path."""
|
||||
tensors: dict[str, torch.Tensor] = {}
|
||||
skeleton = _detach(value, "", tensors)
|
||||
return skeleton, tensors
|
||||
|
||||
|
||||
def attach_cuda_tensors(
|
||||
value: Any,
|
||||
tensors: dict[str, torch.Tensor],
|
||||
device: torch.device | None = None,
|
||||
) -> Any:
|
||||
"""Put detached CUDA tensors back onto the skeleton."""
|
||||
if device is not None:
|
||||
moved: dict[str, torch.Tensor] = {}
|
||||
for key, tensor in tensors.items():
|
||||
moved[key] = (
|
||||
tensor
|
||||
if tensor.device == device
|
||||
else tensor.to(device, non_blocking=True)
|
||||
)
|
||||
tensors = moved
|
||||
return _attach(value, "", tensors)
|
||||
|
||||
|
||||
def _join(prefix: str, part: str) -> str:
|
||||
return part if not prefix else f"{prefix}{_SEP}{part}"
|
||||
|
||||
|
||||
def _detach(value: Any, prefix: str, tensors: dict[str, torch.Tensor]) -> Any:
|
||||
if isinstance(value, torch.Tensor) and value.is_cuda:
|
||||
tensors[prefix] = value
|
||||
return None
|
||||
if isinstance(value, list):
|
||||
return [
|
||||
_detach(item, _join(prefix, str(i)), tensors)
|
||||
for i, item in enumerate(value)
|
||||
]
|
||||
if isinstance(value, tuple):
|
||||
return tuple(
|
||||
_detach(item, _join(prefix, str(i)), tensors)
|
||||
for i, item in enumerate(value)
|
||||
)
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
key: _detach(item, _join(prefix, str(key)), tensors)
|
||||
for key, item in value.items()
|
||||
}
|
||||
if dataclasses.is_dataclass(value) and not isinstance(value, type):
|
||||
value = copy.copy(value)
|
||||
for field in dataclasses.fields(value):
|
||||
current = getattr(value, field.name, None)
|
||||
updated = _detach(current, _join(prefix, field.name), tensors)
|
||||
if updated is not current:
|
||||
setattr(value, field.name, updated)
|
||||
return value
|
||||
return value
|
||||
|
||||
|
||||
def _attach(value: Any, prefix: str, tensors: dict[str, torch.Tensor]) -> Any:
|
||||
if prefix in tensors:
|
||||
return tensors[prefix]
|
||||
if isinstance(value, list):
|
||||
return [
|
||||
_attach(item, _join(prefix, str(i)), tensors)
|
||||
for i, item in enumerate(value)
|
||||
]
|
||||
if isinstance(value, tuple):
|
||||
return tuple(
|
||||
_attach(item, _join(prefix, str(i)), tensors)
|
||||
for i, item in enumerate(value)
|
||||
)
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
key: _attach(item, _join(prefix, str(key)), tensors)
|
||||
for key, item in value.items()
|
||||
}
|
||||
if dataclasses.is_dataclass(value) and not isinstance(value, type):
|
||||
for field in dataclasses.fields(value):
|
||||
current = getattr(value, field.name, None)
|
||||
updated = _attach(current, _join(prefix, field.name), tensors)
|
||||
if updated is not current:
|
||||
setattr(value, field.name, updated)
|
||||
return value
|
||||
return value
|
||||
@@ -0,0 +1,34 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""ComfyUI single-file DiT checkpoints.
|
||||
|
||||
``spec`` holds the shared checkpoint description and load path. Each sibling
|
||||
module registers one DiT family. Importing this package registers every spec.
|
||||
"""
|
||||
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( # noqa: F401
|
||||
flux,
|
||||
minimax_h3,
|
||||
qwen_image,
|
||||
zimage,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
|
||||
ComfyUICheckpointSpec,
|
||||
ParamNamesMapping,
|
||||
WeightIterator,
|
||||
get_comfyui_checkpoint_spec,
|
||||
get_registered_comfyui_pipeline_names,
|
||||
is_comfyui_single_file,
|
||||
load_comfyui_transformer,
|
||||
register_comfyui_checkpoint,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ComfyUICheckpointSpec",
|
||||
"ParamNamesMapping",
|
||||
"WeightIterator",
|
||||
"get_comfyui_checkpoint_spec",
|
||||
"get_registered_comfyui_pipeline_names",
|
||||
"is_comfyui_single_file",
|
||||
"load_comfyui_transformer",
|
||||
"register_comfyui_checkpoint",
|
||||
]
|
||||
@@ -0,0 +1,262 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""ComfyUI Flux checkpoint spec."""
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
|
||||
ComfyUICheckpointSpec,
|
||||
WeightIterator,
|
||||
register_comfyui_checkpoint,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
def _build_dit_config(server_args: ServerArgs) -> FluxConfig:
|
||||
dit_config = getattr(server_args.pipeline_config, "dit_config", None)
|
||||
if not isinstance(dit_config, FluxConfig):
|
||||
dit_config = FluxConfig()
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
|
||||
# ComfyUI Flux checkpoints always carry guidance_in weights.
|
||||
dit_config.arch_config.guidance_embeds = True
|
||||
return dit_config
|
||||
|
||||
|
||||
def _split_sizes(dit_config: FluxConfig) -> tuple[int, int]:
|
||||
arch_config = dit_config.arch_config
|
||||
hidden_size = arch_config.num_attention_heads * arch_config.attention_head_dim
|
||||
mlp_hidden_dim = int(hidden_size * getattr(arch_config, "mlp_ratio", 4.0))
|
||||
return 3 * hidden_size, mlp_hidden_dim
|
||||
|
||||
|
||||
def _split_qkv(
|
||||
tensor: torch.Tensor, hidden_size: int
|
||||
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
|
||||
return (
|
||||
tensor[:hidden_size],
|
||||
tensor[hidden_size : 2 * hidden_size],
|
||||
tensor[2 * hidden_size : 3 * hidden_size],
|
||||
)
|
||||
|
||||
|
||||
def _swap_halves(tensor: torch.Tensor) -> torch.Tensor:
|
||||
"""ComfyUI emits [shift, scale]; AdaLayerNormContinuous expects [scale, shift]."""
|
||||
half = tensor.shape[0] // 2
|
||||
return torch.cat([tensor[half:], tensor[:half]], dim=0)
|
||||
|
||||
|
||||
def _convert_weights(weights: WeightIterator, dit_config: Any) -> WeightIterator:
|
||||
qkv_size, mlp_hidden_dim = _split_sizes(dit_config)
|
||||
has_guidance_embeds = dit_config.arch_config.guidance_embeds
|
||||
hidden_size = qkv_size // 3
|
||||
|
||||
for name, tensor in weights:
|
||||
if not has_guidance_embeds and name.startswith("guidance_in."):
|
||||
continue
|
||||
|
||||
match = re.match(
|
||||
r"double_blocks\.(\d+)\.(img_attn|txt_attn)\.qkv\.(weight|bias)$", name
|
||||
)
|
||||
if match:
|
||||
block_idx, attn_type, param_type = match.groups()
|
||||
if tensor.shape[0] < qkv_size:
|
||||
logger.warning(
|
||||
"%s shape %s smaller than expected qkv size %d, skipping",
|
||||
name,
|
||||
tensor.shape,
|
||||
qkv_size,
|
||||
)
|
||||
continue
|
||||
|
||||
q, k, v = _split_qkv(tensor, hidden_size)
|
||||
prefix = f"transformer_blocks.{block_idx}.attn"
|
||||
if attn_type == "img_attn":
|
||||
yield f"{prefix}.to_q.{param_type}", q
|
||||
yield f"{prefix}.to_k.{param_type}", k
|
||||
yield f"{prefix}.to_v.{param_type}", v
|
||||
else:
|
||||
yield f"{prefix}.add_q_proj.{param_type}", q
|
||||
yield f"{prefix}.add_k_proj.{param_type}", k
|
||||
yield f"{prefix}.add_v_proj.{param_type}", v
|
||||
continue
|
||||
|
||||
match = re.match(r"single_blocks\.(\d+)\.linear1\.(weight|bias)$", name)
|
||||
if match:
|
||||
block_idx, param_type = match.groups()
|
||||
expected_size = qkv_size + mlp_hidden_dim
|
||||
if tensor.shape[0] < expected_size:
|
||||
logger.warning(
|
||||
"linear1.%s shape %s doesn't match expected size %d, skipping",
|
||||
param_type,
|
||||
tensor.shape,
|
||||
expected_size,
|
||||
)
|
||||
continue
|
||||
|
||||
q, k, v = _split_qkv(tensor[:qkv_size], hidden_size)
|
||||
prefix = f"single_transformer_blocks.{block_idx}"
|
||||
yield f"{prefix}.attn.to_q.{param_type}", q
|
||||
yield f"{prefix}.attn.to_k.{param_type}", k
|
||||
yield f"{prefix}.attn.to_v.{param_type}", v
|
||||
yield f"{prefix}.proj_mlp.{param_type}", tensor[qkv_size:]
|
||||
continue
|
||||
|
||||
if name in (
|
||||
"final_layer.adaLN_modulation.1.weight",
|
||||
"final_layer.adaLN_modulation.1.bias",
|
||||
):
|
||||
yield name, _swap_halves(tensor)
|
||||
continue
|
||||
|
||||
yield name, tensor
|
||||
|
||||
|
||||
# ComfyUI names differ from SGLang's diffusers-style names. Fused tensors that
|
||||
# _convert_weights already split are emitted under their SGLang names, so only
|
||||
# the untouched ones need an entry here.
|
||||
_PARAM_NAMES_MAPPING = {
|
||||
r"double_blocks\.(\d+)\.img_attn\.proj\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.attn.to_out.0.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_attn\.proj\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.attn.to_add_out.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_attn\.norm\.query_norm\.scale$": (
|
||||
r"transformer_blocks.\1.attn.norm_q.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_attn\.norm\.key_norm\.scale$": (
|
||||
r"transformer_blocks.\1.attn.norm_k.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_attn\.norm\.query_norm\.scale$": (
|
||||
r"transformer_blocks.\1.attn.norm_added_q.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_attn\.norm\.key_norm\.scale$": (
|
||||
r"transformer_blocks.\1.attn.norm_added_k.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_mlp\.0\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.ff.net.0.proj.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_mlp\.2\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.ff.net.2.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_mlp\.0\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.ff_context.net.0.proj.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_mlp\.2\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.ff_context.net.2.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_mod\.lin\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.norm1.linear.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_mod\.lin\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.norm1_context.linear.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"single_blocks\.(\d+)\.linear2\.(weight|bias)$": (
|
||||
r"single_transformer_blocks.\1.proj_out.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"single_blocks\.(\d+)\.norm\.query_norm\.scale$": (
|
||||
r"single_transformer_blocks.\1.attn.norm_q.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"single_blocks\.(\d+)\.norm\.key_norm\.scale$": (
|
||||
r"single_transformer_blocks.\1.attn.norm_k.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"single_blocks\.(\d+)\.modulation\.lin\.(weight|bias)$": (
|
||||
r"single_transformer_blocks.\1.norm.linear.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^time_in\.in_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.timestep_embedder.linear_1.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^time_in\.out_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.timestep_embedder.linear_2.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^txt_in\.(weight|bias)$": (r"context_embedder.\1", None, None),
|
||||
r"^vector_in\.in_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.text_embedder.linear_1.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^vector_in\.out_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.text_embedder.linear_2.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^final_layer\.linear\.(weight|bias)$": (r"proj_out.\1", None, None),
|
||||
r"^final_layer\.norm_final\.(weight|bias)$": (r"norm_out.\1", None, None),
|
||||
r"^final_layer\.adaLN_modulation\.1\.(weight|bias)$": (
|
||||
r"norm_out.linear.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^img_in\.(weight|bias)$": (r"x_embedder.\1", None, None),
|
||||
r"^guidance_in\.in_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.guidance_embedder.linear_1.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^guidance_in\.out_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.guidance_embedder.linear_2.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
register_comfyui_checkpoint(
|
||||
"FluxPipeline",
|
||||
ComfyUICheckpointSpec(
|
||||
dit_cls_name="FluxTransformer2DModel",
|
||||
build_dit_config=_build_dit_config,
|
||||
param_names_mapping=_PARAM_NAMES_MAPPING,
|
||||
convert_weights=_convert_weights,
|
||||
# ComfyUI Flux checkpoints ship without the optional attention biases.
|
||||
strict=False,
|
||||
# FluxConfig's own mapping reads the same BFL names this spec does, but
|
||||
# targets a different parameter layout (fused to_qkv, ff.linear_in,
|
||||
# time_guidance_embed) than FluxTransformer2DModel exposes. Layering the
|
||||
# two rewrites correctly resolved names into ones the model lacks.
|
||||
inherit_config_mapping=False,
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,35 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""ComfyUI MiniMax-H3 checkpoint spec.
|
||||
|
||||
The Comfy-Org BF16 FL2VA file uses ComfyUI parameter names and already stores
|
||||
fused QKV as ``[q_all, k_all, v_all]``. Identity name mapping is enough, but
|
||||
the DiT's default HF weight loader must not run the grouped→concat reorder.
|
||||
"""
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.minimax_h3 import MiniMaxH3DiTConfig
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
|
||||
ComfyUICheckpointSpec,
|
||||
register_comfyui_checkpoint,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
|
||||
|
||||
def _build_dit_config(server_args: ServerArgs) -> MiniMaxH3DiTConfig:
|
||||
dit_config = getattr(server_args.pipeline_config, "dit_config", None)
|
||||
if not isinstance(dit_config, MiniMaxH3DiTConfig):
|
||||
dit_config = MiniMaxH3DiTConfig()
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
# ComfyUI / Comfy-Org files match the runtime QKV layout. Native
|
||||
# ``sglang serve`` keeps the default grouped HF reorder.
|
||||
dit_config.arch_config.qkv_checkpoint_grouped = False
|
||||
return dit_config
|
||||
|
||||
|
||||
register_comfyui_checkpoint(
|
||||
"MiniMaxH3Pipeline",
|
||||
ComfyUICheckpointSpec(
|
||||
dit_cls_name="MiniMaxH3DiTModel",
|
||||
build_dit_config=_build_dit_config,
|
||||
inherit_config_mapping=True,
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,57 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""ComfyUI Qwen-Image checkpoint specs (text-to-image and edit)."""
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.qwenimage import (
|
||||
QwenImageArchConfig,
|
||||
QwenImageDitConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
|
||||
ComfyUICheckpointSpec,
|
||||
register_comfyui_checkpoint,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
|
||||
|
||||
def _dit_config_builder(
|
||||
zero_cond_t: bool,
|
||||
) -> Callable[[ServerArgs], QwenImageDitConfig]:
|
||||
def build(server_args: ServerArgs) -> QwenImageDitConfig:
|
||||
# ComfyUI checkpoints carry no config, so the architecture is pinned here.
|
||||
dit_config = QwenImageDitConfig(
|
||||
arch_config=QwenImageArchConfig(
|
||||
patch_size=2,
|
||||
in_channels=64,
|
||||
out_channels=16,
|
||||
num_layers=60,
|
||||
attention_head_dim=128,
|
||||
num_attention_heads=24,
|
||||
joint_attention_dim=3584,
|
||||
pooled_projection_dim=768,
|
||||
guidance_embeds=False,
|
||||
axes_dims_rope=(16, 56, 56),
|
||||
zero_cond_t=zero_cond_t,
|
||||
)
|
||||
)
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
return dit_config
|
||||
|
||||
return build
|
||||
|
||||
|
||||
_PARAM_NAMES_MAPPING = {r"^model\.diffusion_model\.(.*)$": (r"\1", None, None)}
|
||||
|
||||
|
||||
for _pipeline_name, _zero_cond_t in (
|
||||
("QwenImagePipeline", False),
|
||||
("QwenImageEditPlusPipeline", True),
|
||||
):
|
||||
register_comfyui_checkpoint(
|
||||
_pipeline_name,
|
||||
ComfyUICheckpointSpec(
|
||||
dit_cls_name="QwenImageTransformer2DModel",
|
||||
build_dit_config=_dit_config_builder(_zero_cond_t),
|
||||
param_names_mapping=_PARAM_NAMES_MAPPING,
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,304 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""ComfyUI single-file DiT checkpoints: spec, registry, and load.
|
||||
|
||||
ComfyUI ships a DiT as one ``.safetensors`` (or a GGUF override) with no
|
||||
``model_index.json`` and its own parameter names. A spec supplies what the
|
||||
shared loader cannot infer: which DiT config to build, how names map onto
|
||||
SGLang, and how to reshape tensors whose layout differs. Per-model specs live
|
||||
in the sibling modules of this package.
|
||||
|
||||
Everything else -- meta-device init, FSDP sharding, quantization, CPU
|
||||
offload -- goes through the regular transformer load path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from collections.abc import Callable, Iterator
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.runtime.distributed import get_local_torch_device
|
||||
from sglang.multimodal_gen.runtime.loader.fsdp_load import maybe_load_fsdp_model
|
||||
from sglang.multimodal_gen.runtime.loader.weight_utils import (
|
||||
safetensors_weights_iterator,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
|
||||
from sglang.multimodal_gen.runtime.utils.precision import resolve_precision
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
ComposedPipelineBase,
|
||||
)
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
WeightIterator = Iterator[tuple[str, torch.Tensor]]
|
||||
|
||||
# ComfyUI mapping entries reuse the param_names_mapping format:
|
||||
# source_regex -> (target_template, merge_index, num_params_to_merge)
|
||||
ParamNamesMapping = dict[str, tuple[str, int | None, int | None]]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ComfyUICheckpointSpec:
|
||||
"""Per-model knowledge needed to load a ComfyUI checkpoint."""
|
||||
|
||||
dit_cls_name: str
|
||||
build_dit_config: Callable[[ServerArgs], Any]
|
||||
param_names_mapping: ParamNamesMapping = field(default_factory=dict)
|
||||
# Reshapes tensors that param_names_mapping cannot express. Receives the raw
|
||||
# safetensors iterator plus the built dit config, yields SGLang-shaped pairs.
|
||||
convert_weights: Callable[[WeightIterator, Any], WeightIterator] | None = None
|
||||
# Set False for checkpoints that legitimately omit parameters the model
|
||||
# declares, such as optional biases.
|
||||
strict: bool = True
|
||||
# Whether to layer param_names_mapping on top of the DiT config's own
|
||||
# mapping. Keep it True when the two act on different names (the config
|
||||
# rules then finish the job, e.g. merging split QKV back into a fused
|
||||
# parameter). Set False when both claim the same source names, since name
|
||||
# mapping is applied repeatedly until it reaches a fixed point and the
|
||||
# config rules would rewrite names this spec already resolved.
|
||||
inherit_config_mapping: bool = True
|
||||
|
||||
|
||||
_SPEC_REGISTRY: dict[str, ComfyUICheckpointSpec] = {}
|
||||
_SPECS_DISCOVERED = False
|
||||
|
||||
|
||||
def register_comfyui_checkpoint(
|
||||
pipeline_name: str, spec: ComfyUICheckpointSpec
|
||||
) -> None:
|
||||
_SPEC_REGISTRY[pipeline_name] = spec
|
||||
|
||||
|
||||
def _discover_checkpoint_specs() -> None:
|
||||
global _SPECS_DISCOVERED
|
||||
if _SPECS_DISCOVERED:
|
||||
return
|
||||
_SPECS_DISCOVERED = True
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import ( # noqa: F401
|
||||
flux,
|
||||
minimax_h3,
|
||||
qwen_image,
|
||||
zimage,
|
||||
)
|
||||
|
||||
|
||||
def get_comfyui_checkpoint_spec(pipeline_name: str) -> ComfyUICheckpointSpec | None:
|
||||
_discover_checkpoint_specs()
|
||||
return _SPEC_REGISTRY.get(pipeline_name)
|
||||
|
||||
|
||||
def get_registered_comfyui_pipeline_names() -> list[str]:
|
||||
_discover_checkpoint_specs()
|
||||
return sorted(_SPEC_REGISTRY)
|
||||
|
||||
|
||||
def is_comfyui_single_file(model_path: str) -> bool:
|
||||
"""ComfyUI ships DiTs as one safetensors file with no model_index.json."""
|
||||
return os.path.isfile(model_path) and model_path.endswith(".safetensors")
|
||||
|
||||
|
||||
def _load_comfyui_gguf_transformer(
|
||||
*,
|
||||
model_cls,
|
||||
dit_config,
|
||||
gguf_file: str,
|
||||
server_args: ServerArgs,
|
||||
mapping: dict[str, Any],
|
||||
spec,
|
||||
):
|
||||
from sglang.multimodal_gen.runtime.layers.quantization.gguf import GGUFConfig
|
||||
from sglang.multimodal_gen.runtime.loader.gguf_weights import (
|
||||
gguf_weights_iterator,
|
||||
read_gguf_tensor_meta,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.minimax_h3_weights import (
|
||||
validate_minimax_h3_checkpoint_variant,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.transformer_load_utils import (
|
||||
_validate_gguf_runtime_support,
|
||||
)
|
||||
|
||||
_validate_gguf_runtime_support(server_args, "transformer")
|
||||
if spec.dit_cls_name == "MiniMaxH3DiTModel":
|
||||
validate_minimax_h3_checkpoint_variant(
|
||||
[gguf_file], str(server_args.model_variant or "fl2va")
|
||||
)
|
||||
|
||||
tensor_meta = read_gguf_tensor_meta(gguf_file)
|
||||
curve = tensor_meta.get("adaln_t_table")
|
||||
if curve is not None:
|
||||
if curve.is_quantized or len(curve.logical_shape) != 2:
|
||||
raise ValueError(
|
||||
"MiniMax-H3 adaln_t_table must be an unquantized 2D tensor"
|
||||
)
|
||||
curve_grid, time_embed_dim = curve.logical_shape
|
||||
if curve_grid < 2:
|
||||
raise ValueError("MiniMax-H3 adaln_t_table needs at least two rows")
|
||||
dit_config.arch_config.adaln_curve_grid = curve_grid
|
||||
dit_config.arch_config.time_embed_dim = time_embed_dim
|
||||
|
||||
quant_config = GGUFConfig(gguf_file=gguf_file, tensor_meta=tensor_meta)
|
||||
packed = getattr(model_cls, "packed_modules_mapping", None)
|
||||
if packed:
|
||||
quant_config.packed_modules_mapping = packed
|
||||
|
||||
logger.info("Loading %s from GGUF file %s", spec.dit_cls_name, gguf_file)
|
||||
original_mapping = model_cls.param_names_mapping
|
||||
model_cls.param_names_mapping = mapping
|
||||
try:
|
||||
return maybe_load_fsdp_model(
|
||||
model_cls=model_cls,
|
||||
init_params={
|
||||
"config": dit_config,
|
||||
"hf_config": {},
|
||||
"quant_config": quant_config,
|
||||
},
|
||||
weight_dir_list=[],
|
||||
device=get_local_torch_device(),
|
||||
hsdp_replicate_dim=server_args.hsdp_replicate_dim,
|
||||
hsdp_shard_dim=server_args.hsdp_shard_dim,
|
||||
component_starts_on_cpu=server_args.should_start_component_on_cpu(
|
||||
"transformer"
|
||||
),
|
||||
pin_cpu_memory=server_args.pin_cpu_memory,
|
||||
fsdp_inference=server_args.should_use_fsdp_for_component("transformer"),
|
||||
param_dtype=None,
|
||||
reduce_dtype=torch.float32,
|
||||
output_dtype=None,
|
||||
strict=False,
|
||||
weights_iterator=gguf_weights_iterator(gguf_file, tensor_meta),
|
||||
)
|
||||
finally:
|
||||
model_cls.param_names_mapping = original_mapping
|
||||
|
||||
|
||||
def load_comfyui_transformer(
|
||||
pipeline: ComposedPipelineBase,
|
||||
server_args: ServerArgs,
|
||||
loaded_modules: dict[str, torch.nn.Module] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Load the DiT from a single ComfyUI safetensors file.
|
||||
|
||||
Reuses the shared FSDP-aware transformer load path; the spec only supplies
|
||||
what the file itself cannot describe. A GGUF
|
||||
``--transformer-weights-path`` replaces only the DiT weights.
|
||||
"""
|
||||
if loaded_modules is not None and "transformer" in loaded_modules:
|
||||
return {
|
||||
"transformer": loaded_modules["transformer"],
|
||||
"scheduler": pipeline.get_module("scheduler"),
|
||||
}
|
||||
|
||||
spec = get_comfyui_checkpoint_spec(pipeline.pipeline_name)
|
||||
if spec is None:
|
||||
raise ValueError(
|
||||
f"{pipeline.pipeline_name} has no ComfyUI checkpoint spec, so it cannot "
|
||||
f"load a single safetensors file. Pipelines with a spec: "
|
||||
f"{get_registered_comfyui_pipeline_names()}"
|
||||
)
|
||||
|
||||
model_path = pipeline.model_path
|
||||
dit_config = spec.build_dit_config(server_args)
|
||||
mapping = dict(spec.param_names_mapping)
|
||||
if spec.inherit_config_mapping:
|
||||
mapping = {
|
||||
**(dit_config.arch_config.param_names_mapping or {}),
|
||||
**mapping,
|
||||
}
|
||||
dit_config.arch_config.param_names_mapping = mapping
|
||||
|
||||
model_cls, _ = ModelRegistry.resolve_model_cls(spec.dit_cls_name)
|
||||
param_dtype = resolve_precision(server_args, "dit", precision_attr="dit_precision")
|
||||
server_args.model_paths["transformer"] = os.path.dirname(model_path) or "."
|
||||
|
||||
from sglang.multimodal_gen.runtime.loader.transformer_load_utils import (
|
||||
resolve_transformer_gguf_to_load,
|
||||
)
|
||||
|
||||
gguf_file = resolve_transformer_gguf_to_load(server_args, "transformer")
|
||||
if gguf_file is not None:
|
||||
model = _load_comfyui_gguf_transformer(
|
||||
model_cls=model_cls,
|
||||
dit_config=dit_config,
|
||||
gguf_file=gguf_file,
|
||||
server_args=server_args,
|
||||
mapping=mapping,
|
||||
spec=spec,
|
||||
)
|
||||
else:
|
||||
# Only override the iterator when tensors need reshaping; leaving it None
|
||||
# keeps the rank-local checkpoint fast path available.
|
||||
weights_iterator = None
|
||||
if spec.convert_weights is not None:
|
||||
weights_iterator = spec.convert_weights(
|
||||
safetensors_weights_iterator([model_path]), dit_config
|
||||
)
|
||||
|
||||
# GGUF already sets AdaLN curve from tensor meta. Pruned BF16
|
||||
# safetensors keep the same adaln_t_table; without this the DiT is
|
||||
# built as the unpruned MLP and load fails on that extra parameter.
|
||||
if spec.dit_cls_name == "MiniMaxH3DiTModel":
|
||||
from sglang.multimodal_gen.runtime.loader.minimax_h3_weights import (
|
||||
inspect_minimax_h3_safetensors,
|
||||
)
|
||||
|
||||
adaln_curve_shape, _ = inspect_minimax_h3_safetensors([model_path])
|
||||
if adaln_curve_shape is not None:
|
||||
(
|
||||
dit_config.arch_config.adaln_curve_grid,
|
||||
dit_config.arch_config.time_embed_dim,
|
||||
) = adaln_curve_shape
|
||||
logger.info(
|
||||
"MiniMax-H3 ComfyUI checkpoint uses AdaLN curve %s",
|
||||
adaln_curve_shape,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Loading %s from ComfyUI checkpoint %s, param_dtype: %s",
|
||||
spec.dit_cls_name,
|
||||
model_path,
|
||||
param_dtype,
|
||||
)
|
||||
|
||||
# Weight loading reads param_names_mapping off the model, which inherits it
|
||||
# from the class, so the ComfyUI names have to be visible for the whole load.
|
||||
original_mapping = model_cls.param_names_mapping
|
||||
model_cls.param_names_mapping = mapping
|
||||
try:
|
||||
model = maybe_load_fsdp_model(
|
||||
model_cls=model_cls,
|
||||
init_params={"config": dit_config, "hf_config": {}},
|
||||
weight_dir_list=[model_path],
|
||||
device=get_local_torch_device(),
|
||||
hsdp_replicate_dim=server_args.hsdp_replicate_dim,
|
||||
hsdp_shard_dim=server_args.hsdp_shard_dim,
|
||||
component_starts_on_cpu=server_args.should_start_component_on_cpu(
|
||||
"transformer"
|
||||
),
|
||||
pin_cpu_memory=server_args.pin_cpu_memory,
|
||||
fsdp_inference=server_args.should_use_fsdp_for_component("transformer"),
|
||||
param_dtype=param_dtype,
|
||||
reduce_dtype=torch.float32,
|
||||
output_dtype=None,
|
||||
strict=spec.strict,
|
||||
weights_iterator=weights_iterator,
|
||||
)
|
||||
finally:
|
||||
model_cls.param_names_mapping = original_mapping
|
||||
|
||||
for param in model.parameters():
|
||||
param.requires_grad = False
|
||||
|
||||
logger.info(
|
||||
"Loaded transformer with %.2fB parameters",
|
||||
sum(p.numel() for p in model.parameters()) / 1e9,
|
||||
)
|
||||
|
||||
return {"transformer": model, "scheduler": pipeline.get_module("scheduler")}
|
||||
@@ -0,0 +1,65 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""ComfyUI Z-Image (lumina2) checkpoint spec."""
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.zimage import ZImageDitConfig
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints.spec import (
|
||||
ComfyUICheckpointSpec,
|
||||
WeightIterator,
|
||||
register_comfyui_checkpoint,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
|
||||
|
||||
def _build_dit_config(server_args: ServerArgs) -> ZImageDitConfig:
|
||||
dit_config = getattr(server_args.pipeline_config, "dit_config", None)
|
||||
if not isinstance(dit_config, ZImageDitConfig):
|
||||
dit_config = ZImageDitConfig()
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
return dit_config
|
||||
|
||||
|
||||
def _convert_weights(weights: WeightIterator, dit_config: Any) -> WeightIterator:
|
||||
"""Split ComfyUI's merged attention qkv into to_q / to_k / to_v."""
|
||||
arch_config = dit_config.arch_config
|
||||
q_size = arch_config.dim
|
||||
k_size = (
|
||||
arch_config.dim // arch_config.num_attention_heads
|
||||
) * arch_config.n_kv_heads
|
||||
|
||||
for name, tensor in weights:
|
||||
match = re.match(
|
||||
r"(layers|noise_refiner|context_refiner)\.(\d+)\.attention\.qkv\.(weight|bias)$",
|
||||
name,
|
||||
)
|
||||
if not match:
|
||||
yield name, tensor
|
||||
continue
|
||||
|
||||
module_name, layer_idx, param_type = match.groups()
|
||||
prefix = f"{module_name}.{layer_idx}.attention"
|
||||
yield f"{prefix}.to_q.{param_type}", tensor[:q_size]
|
||||
yield f"{prefix}.to_k.{param_type}", tensor[q_size : q_size + k_size]
|
||||
yield f"{prefix}.to_v.{param_type}", tensor[q_size + k_size :]
|
||||
|
||||
|
||||
_PARAM_NAMES_MAPPING = {
|
||||
r"(.*)\.attention\.k_norm\.weight$": (r"\1.attention.norm_k.weight", None, None),
|
||||
r"(.*)\.attention\.q_norm\.weight$": (r"\1.attention.norm_q.weight", None, None),
|
||||
r"(.*)\.attention\.out\.weight$": (r"\1.attention.to_out.0.weight", None, None),
|
||||
r"^final_layer\.(.*)$": (r"all_final_layer.2-1.\1", None, None),
|
||||
r"^x_embedder\.(.*)$": (r"all_x_embedder.2-1.\1", None, None),
|
||||
}
|
||||
|
||||
|
||||
register_comfyui_checkpoint(
|
||||
"ZImagePipeline",
|
||||
ComfyUICheckpointSpec(
|
||||
dit_cls_name="ZImageTransformer2DModel",
|
||||
build_dit_config=_build_dit_config,
|
||||
param_names_mapping=_PARAM_NAMES_MAPPING,
|
||||
convert_weights=_convert_weights,
|
||||
),
|
||||
)
|
||||
@@ -15,6 +15,7 @@ from torch.distributed.fsdp import FSDPModule
|
||||
from sglang.multimodal_gen.runtime.distributed import get_tp_rank, get_tp_world_size
|
||||
from sglang.multimodal_gen.runtime.layers.linear import (
|
||||
ColumnParallelLinear,
|
||||
MergedColumnParallelLinear,
|
||||
ReplicatedLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
@@ -236,6 +237,17 @@ def read_rank_local_tensor(
|
||||
return local_tensor
|
||||
|
||||
|
||||
def _merged_output_sizes(actual_param: torch.nn.Parameter) -> list[int] | None:
|
||||
"""Logical output widths of a fused-on-disk MergedColumnParallelLinear."""
|
||||
weight_loader = actual_param.__dict__.get("weight_loader")
|
||||
if not isinstance(weight_loader, MethodType):
|
||||
return None
|
||||
owner = weight_loader.__self__
|
||||
if isinstance(owner, MergedColumnParallelLinear):
|
||||
return list(owner.output_sizes)
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_tp_shard_dim(
|
||||
actual_param: torch.nn.Parameter,
|
||||
) -> tuple[bool, int | None]:
|
||||
@@ -318,6 +330,7 @@ def read_tp_local_tensor(
|
||||
shard_dim: int | None,
|
||||
tp_rank: int,
|
||||
tp_size: int,
|
||||
output_sizes: list[int] | None = None,
|
||||
) -> torch.Tensor:
|
||||
if shard_dim is None:
|
||||
assembled_shape = assembled_source_shape(sources)
|
||||
@@ -337,6 +350,37 @@ def read_tp_local_tensor(
|
||||
source.merge_index if source.merge_index is not None else 0,
|
||||
),
|
||||
)
|
||||
# Fused-on-disk [Q|K|V] / [gate|up]: each logical matrix is sharded on
|
||||
# its own, then concatenated. A contiguous half of the fused tensor is
|
||||
# the same shape as the TP-local parameter but the wrong values.
|
||||
if (
|
||||
output_sizes is not None
|
||||
and len(ordered_sources) == 1
|
||||
and all(size % tp_size == 0 for size in output_sizes)
|
||||
):
|
||||
source = ordered_sources[0]
|
||||
if source.shape[shard_dim] != sum(output_sizes):
|
||||
raise RuntimeError(
|
||||
"Merged TP shard sizes do not cover the fused checkpoint: "
|
||||
f"output_sizes={output_sizes}, source_dim={source.shape[shard_dim]}"
|
||||
)
|
||||
local_parts = []
|
||||
offset = 0
|
||||
for output_size in output_sizes:
|
||||
shard_size = output_size // tp_size
|
||||
slices = [slice(None)] * len(source.shape)
|
||||
slices[shard_dim] = slice(
|
||||
offset + tp_rank * shard_size,
|
||||
offset + tp_rank * shard_size + shard_size,
|
||||
)
|
||||
local_parts.append(
|
||||
handles[source.file_path]
|
||||
.get_slice(source.param_name)[tuple(slices)]
|
||||
.contiguous()
|
||||
)
|
||||
offset += output_size
|
||||
return torch.cat(local_parts, dim=shard_dim)
|
||||
|
||||
local_parts = []
|
||||
for source in ordered_sources:
|
||||
shard_size = source.shape[shard_dim] // tp_size
|
||||
@@ -380,6 +424,7 @@ def try_load_rank_local_tp_state_dict(
|
||||
sources_by_target, reverse_param_names_mapping = checkpoint_sources
|
||||
|
||||
shard_dims: dict[str, int | None] = {}
|
||||
output_sizes_by_name: dict[str, list[int] | None] = {}
|
||||
for target_param_name, sources in sources_by_target.items():
|
||||
meta_param = meta_sd.get(target_param_name)
|
||||
if meta_param is None or isinstance(meta_param, dist_tensor.DTensor):
|
||||
@@ -394,15 +439,22 @@ def try_load_rank_local_tp_state_dict(
|
||||
param_dict,
|
||||
target_param_name,
|
||||
)
|
||||
output_sizes = None
|
||||
if actual_param is None:
|
||||
supported, shard_dim = True, None
|
||||
else:
|
||||
supported, shard_dim = _resolve_tp_shard_dim(actual_param)
|
||||
output_sizes = _merged_output_sizes(actual_param)
|
||||
if output_sizes is not None and any(
|
||||
size % tp_size for size in output_sizes
|
||||
):
|
||||
return None
|
||||
if not supported:
|
||||
return None
|
||||
if tp_local_shape(sources, shard_dim, tp_size) != tuple(meta_param.shape):
|
||||
return None
|
||||
shard_dims[target_param_name] = shard_dim
|
||||
output_sizes_by_name[target_param_name] = output_sizes
|
||||
|
||||
local_param_sd: dict[str, LocalTPShard] = {}
|
||||
local_bytes = 0
|
||||
@@ -421,6 +473,7 @@ def try_load_rank_local_tp_state_dict(
|
||||
shard_dims[target_param_name],
|
||||
tp_rank,
|
||||
tp_size,
|
||||
output_sizes=output_sizes_by_name[target_param_name],
|
||||
)
|
||||
local_param_sd[target_param_name] = LocalTPShard(tensor)
|
||||
local_bytes += tensor.numel() * tensor.element_size()
|
||||
|
||||
@@ -16,6 +16,11 @@ from sglang.multimodal_gen.runtime.disaggregation.roles import RoleType
|
||||
from sglang.multimodal_gen.runtime.disaggregation.scheduler_mixin import (
|
||||
SchedulerDisaggMixin,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.distributed.ipc_cuda import (
|
||||
materialize_cuda_refs,
|
||||
release_retained_producer_tensors,
|
||||
spill_cuda_tensors,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.distributed.utils import broadcast_pyobj
|
||||
from sglang.multimodal_gen.runtime.entrypoints.control_requests import (
|
||||
GetDisaggStatsReq,
|
||||
@@ -771,6 +776,13 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag
|
||||
output_batch.output
|
||||
)
|
||||
|
||||
with self._record_return_stage(
|
||||
output_batch, "Scheduler.return_result.spill_cuda"
|
||||
):
|
||||
# The previous reply has already been mapped: the client
|
||||
# only sends the next hop after materializing the last one.
|
||||
release_retained_producer_tensors()
|
||||
spill_cuda_tensors(output_batch, in_place=True)
|
||||
with self._record_return_stage(
|
||||
output_batch, "Scheduler.return_result.pickle"
|
||||
):
|
||||
@@ -1183,7 +1195,9 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag
|
||||
|
||||
def recv_reqs(self) -> List[tuple[bytes, Any]]:
|
||||
"""
|
||||
For non-main schedulers, reqs are broadcasted from main using broadcast_pyobj
|
||||
For non-main schedulers, reqs are broadcasted from main using
|
||||
broadcast_pyobj. ``--comfyui-mode`` multi-rank instead keeps CUDA
|
||||
tensors on NCCL so per-step latents do not pickle onto the gloo group.
|
||||
"""
|
||||
if self.receiver is not None:
|
||||
try:
|
||||
@@ -1208,30 +1222,37 @@ class Scheduler(SchedulerWarmupMixin, SchedulerPostTrainingMixin, SchedulerDisag
|
||||
else:
|
||||
recv_reqs = None
|
||||
|
||||
# TODO: fix this condition
|
||||
if self.server_args.sp_degree != 1:
|
||||
recv_reqs = broadcast_pyobj(
|
||||
recv_reqs,
|
||||
self.worker.sp_group.rank,
|
||||
self.worker.sp_cpu_group,
|
||||
src=self.worker.sp_group.ranks[0],
|
||||
)
|
||||
# Rebuild CUDA IPC handles on rank 0 (no-op when the payload has none).
|
||||
if recv_reqs is not None:
|
||||
recv_reqs = materialize_cuda_refs(recv_reqs)
|
||||
|
||||
if self.server_args.enable_cfg_parallel:
|
||||
recv_reqs = broadcast_pyobj(
|
||||
recv_reqs,
|
||||
self.worker.cfg_group.rank,
|
||||
self.worker.cfg_cpu_group,
|
||||
src=self.worker.cfg_group.ranks[0],
|
||||
)
|
||||
if self.server_args.comfyui_mode and self._is_multi_rank():
|
||||
recv_reqs = self._broadcast_recv_reqs(recv_reqs)
|
||||
else:
|
||||
# TODO: fix this condition
|
||||
if self.server_args.sp_degree != 1:
|
||||
recv_reqs = broadcast_pyobj(
|
||||
recv_reqs,
|
||||
self.worker.sp_group.rank,
|
||||
self.worker.sp_cpu_group,
|
||||
src=self.worker.sp_group.ranks[0],
|
||||
)
|
||||
|
||||
if self.server_args.tp_size > 1:
|
||||
recv_reqs = broadcast_pyobj(
|
||||
recv_reqs,
|
||||
self.worker.tp_group.rank,
|
||||
self.worker.tp_cpu_group,
|
||||
src=self.worker.tp_group.ranks[0],
|
||||
)
|
||||
if self.server_args.enable_cfg_parallel:
|
||||
recv_reqs = broadcast_pyobj(
|
||||
recv_reqs,
|
||||
self.worker.cfg_group.rank,
|
||||
self.worker.cfg_cpu_group,
|
||||
src=self.worker.cfg_group.ranks[0],
|
||||
)
|
||||
|
||||
if self.server_args.tp_size > 1:
|
||||
recv_reqs = broadcast_pyobj(
|
||||
recv_reqs,
|
||||
self.worker.tp_group.rank,
|
||||
self.worker.tp_cpu_group,
|
||||
src=self.worker.tp_group.ranks[0],
|
||||
)
|
||||
|
||||
assert recv_reqs is not None
|
||||
|
||||
|
||||
@@ -822,8 +822,9 @@ class MiniMaxH3Attention(nn.Module):
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.qkv_proj",
|
||||
)
|
||||
# Official safetensors interleave Q/K/V by head. Comfy and GGUF
|
||||
# checkpoints already store [q_all, k_all, v_all].
|
||||
# Official safetensors interleave Q/K/V by head. Comfy BF16, Comfy
|
||||
# convrot int8, and GGUF already store [q_all, k_all, v_all].
|
||||
skip_grouped_reorder = not getattr(arch, "qkv_checkpoint_grouped", True)
|
||||
checkpoint_qkv_is_native = quant_config is not None and (
|
||||
quant_config.get_name() == "gguf"
|
||||
or quant_config.checkpoint_uses_native_qkv_layout
|
||||
@@ -831,7 +832,7 @@ class MiniMaxH3Attention(nn.Module):
|
||||
checkpoint_qkv_is_native = (
|
||||
checkpoint_qkv_is_native or arch.checkpoint_uses_diffusers_layout
|
||||
)
|
||||
if not checkpoint_qkv_is_native:
|
||||
if not skip_grouped_reorder and not checkpoint_qkv_is_native:
|
||||
self._install_qkv_weight_loader(arch)
|
||||
self.q_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
|
||||
self.k_norm = _norm(arch.attention_head_dim, eps=arch.qk_norm_eps)
|
||||
|
||||
+3
-1
@@ -112,7 +112,9 @@ class ComfyUIPassThroughScheduler(BaseScheduler, ConfigMixin, SchedulerMixin):
|
||||
Returns:
|
||||
The input sample unchanged (prev_sample = sample)
|
||||
"""
|
||||
# Increment step index for tracking
|
||||
# DenoisingStage resets _step_index to None before each request.
|
||||
if self._step_index is None:
|
||||
self._step_index = 0
|
||||
self._step_index += 1
|
||||
|
||||
# Simply return the input sample unchanged
|
||||
|
||||
@@ -1,701 +0,0 @@
|
||||
# Copied and adapted from: https://github.com/hao-ai-lab/FastVideo
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import os
|
||||
import re
|
||||
from typing import Any, Generator
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig
|
||||
from sglang.multimodal_gen.runtime.distributed import get_local_torch_device
|
||||
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
|
||||
from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import (
|
||||
ComfyUIPassThroughScheduler,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
ComposedPipelineBase,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages import (
|
||||
ComfyUILatentPreparationStage,
|
||||
DenoisingStage,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
|
||||
from sglang.multimodal_gen.runtime.utils.precision import resolve_precision
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
class ComfyUIFluxPipeline(LoRAPipeline, ComposedPipelineBase):
|
||||
"""
|
||||
Simplified pipeline for ComfyUI integration with only denoising stage.
|
||||
|
||||
This pipeline requires pre-processed inputs:
|
||||
- prompt_embeds: Pre-encoded text embeddings (list of tensors)
|
||||
- negative_prompt_embeds: Pre-encoded negative prompt embeddings (if using CFG)
|
||||
- latents: Optional initial noise latents (will be generated if not provided)
|
||||
|
||||
Usage:
|
||||
generator = DiffGenerator.from_pretrained(
|
||||
model_path="path/to/model",
|
||||
pipeline_class_name="ComfyUIFluxPipeline",
|
||||
device="cuda",
|
||||
)
|
||||
"""
|
||||
|
||||
pipeline_name = "ComfyUIFluxPipeline"
|
||||
|
||||
# Configuration classes for safetensors files without model_index.json
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig
|
||||
from sglang.multimodal_gen.configs.sample.flux import FluxSamplingParams
|
||||
|
||||
pipeline_config_cls = FluxPipelineConfig
|
||||
sampling_params_cls = FluxSamplingParams
|
||||
|
||||
_required_config_modules = [
|
||||
"transformer",
|
||||
"scheduler",
|
||||
]
|
||||
|
||||
def initialize_pipeline(self, server_args: ServerArgs):
|
||||
"""
|
||||
Initialize the pipeline with ComfyUI pass-through scheduler.
|
||||
This scheduler does not modify latents, allowing ComfyUI to handle denoising.
|
||||
"""
|
||||
self.modules["scheduler"] = ComfyUIPassThroughScheduler(
|
||||
num_train_timesteps=1000
|
||||
)
|
||||
|
||||
if hasattr(server_args.pipeline_config, "vae_config"):
|
||||
vae_config = server_args.pipeline_config.vae_config
|
||||
if hasattr(vae_config, "post_init") and not hasattr(
|
||||
vae_config, "_post_init_called"
|
||||
):
|
||||
vae_config.post_init()
|
||||
logger.info(
|
||||
"Called vae_config.post_init() to set spatial_compression_ratio. "
|
||||
f"spatial_compression_ratio={vae_config.arch_config.spatial_compression_ratio}"
|
||||
)
|
||||
|
||||
def load_modules(
|
||||
self,
|
||||
server_args: ServerArgs,
|
||||
loaded_modules: dict[str, torch.nn.Module] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Load modules for ComfyUIFluxPipeline.
|
||||
|
||||
If model_path is a safetensors file, load transformer directly from it
|
||||
without requiring model_index.json. Otherwise, fall back to default loading.
|
||||
"""
|
||||
if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"):
|
||||
logger.info(
|
||||
"Detected safetensors file, loading transformer directly from: %s",
|
||||
self.model_path,
|
||||
)
|
||||
return self._load_transformer_from_safetensors(server_args, loaded_modules)
|
||||
else:
|
||||
logger.info(
|
||||
"Model path is a directory, using default loading method: %s",
|
||||
self.model_path,
|
||||
)
|
||||
return super().load_modules(server_args, loaded_modules)
|
||||
|
||||
def _load_and_convert_weights_from_safetensors(
|
||||
self,
|
||||
model_cls: type,
|
||||
dit_config: FluxConfig,
|
||||
hf_config: dict,
|
||||
safetensors_list: list[str],
|
||||
updated_mapping: dict,
|
||||
qkv_size: int,
|
||||
mlp_hidden_dim: int,
|
||||
has_guidance_embeds: bool,
|
||||
default_dtype: torch.dtype,
|
||||
) -> tuple[torch.nn.Module, dict]:
|
||||
"""
|
||||
Load and convert weights from safetensors file, then load them into the model.
|
||||
"""
|
||||
from sglang.multimodal_gen.runtime.loader.utils import (
|
||||
get_param_names_mapping,
|
||||
set_default_torch_dtype,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.weight_utils import (
|
||||
safetensors_weights_iterator,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Converting ComfyUI Flux weights to SGLang format and loading model..."
|
||||
)
|
||||
|
||||
# Create model on target device
|
||||
device = get_local_torch_device()
|
||||
with set_default_torch_dtype(default_dtype):
|
||||
model = model_cls(**{"config": dit_config, "hf_config": hf_config})
|
||||
model = model.to(device)
|
||||
|
||||
# Verify model has guidance_embedder if config says it should
|
||||
has_guidance_embedder = hasattr(model.time_text_embed, "guidance_embedder")
|
||||
if has_guidance_embeds and not has_guidance_embedder:
|
||||
logger.warning(
|
||||
"Config has guidance_embeds=True but model doesn't have guidance_embedder. "
|
||||
"This may indicate a configuration mismatch."
|
||||
)
|
||||
elif not has_guidance_embeds and has_guidance_embedder:
|
||||
logger.warning(
|
||||
"Config has guidance_embeds=False but model has guidance_embedder. "
|
||||
"This may indicate a configuration mismatch."
|
||||
)
|
||||
|
||||
# Note: guidance_in mappings are already included in comfyui_flux_mappings above.
|
||||
# If model doesn't support guidance embeddings, the weights will be filtered out
|
||||
# in _convert_comfyui_weights() based on has_guidance_embeds flag.
|
||||
|
||||
param_names_mapping_fn = get_param_names_mapping(updated_mapping)
|
||||
|
||||
weight_iterator = safetensors_weights_iterator(safetensors_list)
|
||||
converted_weights = self._convert_comfyui_weights(
|
||||
weight_iterator=weight_iterator,
|
||||
qkv_size=qkv_size,
|
||||
mlp_hidden_dim=mlp_hidden_dim,
|
||||
has_guidance_embeds=has_guidance_embeds,
|
||||
)
|
||||
|
||||
model_state_dict = model.state_dict()
|
||||
missing_keys = set(model_state_dict.keys())
|
||||
unexpected_keys = []
|
||||
loaded_count = 0
|
||||
reverse_param_names_mapping = {}
|
||||
|
||||
# Handle merged parameters (collect all parts before merging)
|
||||
from collections import defaultdict
|
||||
|
||||
to_merge_params = defaultdict(dict)
|
||||
|
||||
# Process weights incrementally: load immediately after conversion
|
||||
for source_name, tensor in converted_weights:
|
||||
target_name, merge_index, num_params_to_merge = param_names_mapping_fn(
|
||||
source_name
|
||||
)
|
||||
reverse_param_names_mapping[target_name] = (
|
||||
source_name,
|
||||
merge_index,
|
||||
num_params_to_merge,
|
||||
)
|
||||
|
||||
if merge_index is not None:
|
||||
# Collect parts for merging
|
||||
to_merge_params[target_name][merge_index] = tensor
|
||||
if len(to_merge_params[target_name]) == num_params_to_merge:
|
||||
# All parts collected, merge them
|
||||
sorted_tensors = [
|
||||
to_merge_params[target_name][i]
|
||||
for i in range(num_params_to_merge)
|
||||
]
|
||||
merged_tensor = torch.cat(sorted_tensors, dim=0)
|
||||
# Load immediately after merging
|
||||
if target_name in model_state_dict:
|
||||
param = model_state_dict[target_name]
|
||||
loaded_tensor = merged_tensor.to(
|
||||
device=param.device, dtype=param.dtype
|
||||
)
|
||||
param.data.copy_(loaded_tensor)
|
||||
missing_keys.discard(target_name)
|
||||
loaded_count += 1
|
||||
del merged_tensor, loaded_tensor
|
||||
else:
|
||||
unexpected_keys.append(target_name)
|
||||
# Clear merged parts
|
||||
del to_merge_params[target_name]
|
||||
for t in sorted_tensors:
|
||||
del t
|
||||
else:
|
||||
# Direct mapping, load immediately
|
||||
if target_name in model_state_dict:
|
||||
param = model_state_dict[target_name]
|
||||
# Check shape compatibility
|
||||
if tensor.shape != param.shape:
|
||||
logger.warning(
|
||||
f"Shape mismatch for {target_name}: "
|
||||
f"loaded {tensor.shape} vs model {param.shape}, skipping. "
|
||||
f"Source: {source_name}"
|
||||
)
|
||||
unexpected_keys.append(target_name)
|
||||
del tensor
|
||||
continue
|
||||
|
||||
# Debug logging for norm_out.linear to verify mapping
|
||||
if (
|
||||
"norm_out.linear" in target_name
|
||||
or "final_layer.adaLN_modulation" in source_name
|
||||
):
|
||||
logger.info(
|
||||
f"Loading norm_out.linear: {source_name} -> {target_name}, "
|
||||
f"shape: {tensor.shape}"
|
||||
)
|
||||
|
||||
loaded_tensor = tensor.to(device=param.device, dtype=param.dtype)
|
||||
param.data.copy_(loaded_tensor)
|
||||
missing_keys.discard(target_name)
|
||||
loaded_count += 1
|
||||
del tensor, loaded_tensor
|
||||
else:
|
||||
# Debug logging for unmapped parameters
|
||||
if "norm_out.linear" in target_name:
|
||||
logger.warning(
|
||||
f"norm_out.linear parameter {target_name} not found in model state_dict. "
|
||||
f"Source: {source_name}"
|
||||
)
|
||||
unexpected_keys.append(target_name)
|
||||
|
||||
optional_missing_keys = []
|
||||
required_missing_keys = []
|
||||
for key in missing_keys:
|
||||
if key.endswith(".bias"):
|
||||
# Check if corresponding weight exists (if weight exists but bias doesn't, it's optional)
|
||||
weight_key = key.replace(".bias", ".weight")
|
||||
if weight_key not in missing_keys:
|
||||
optional_missing_keys.append(key)
|
||||
else:
|
||||
required_missing_keys.append(key)
|
||||
else:
|
||||
required_missing_keys.append(key)
|
||||
|
||||
if required_missing_keys:
|
||||
logger.warning(
|
||||
f"Required missing keys (first 10): {required_missing_keys[:10]}..."
|
||||
)
|
||||
if optional_missing_keys:
|
||||
logger.info(
|
||||
f"Optional missing keys (bias parameters, {len(optional_missing_keys)} total): "
|
||||
f"These will use default values (zeros)"
|
||||
)
|
||||
if unexpected_keys:
|
||||
logger.warning(f"Unexpected keys (first 10): {unexpected_keys[:10]}...")
|
||||
|
||||
logger.info(f"Successfully loaded {loaded_count} weight tensors")
|
||||
|
||||
return model, reverse_param_names_mapping
|
||||
|
||||
def _convert_comfyui_weights(
|
||||
self,
|
||||
weight_iterator: Generator[tuple[str, torch.Tensor], None, None],
|
||||
qkv_size: int,
|
||||
mlp_hidden_dim: int,
|
||||
has_guidance_embeds: bool,
|
||||
) -> Generator[tuple[str, torch.Tensor], None, None]:
|
||||
"""
|
||||
Convert ComfyUI Flux weights to SGLang format.
|
||||
Splits fused qkv weights into to_q/to_k/to_v plus proj_mlp.
|
||||
Filters out guidance_in weights if model doesn't support guidance embeddings.
|
||||
Handles scale/shift order difference between ComfyUI and AdaLayerNormContinuous.
|
||||
"""
|
||||
for name, tensor in weight_iterator:
|
||||
if not has_guidance_embeds and name.startswith("guidance_in."):
|
||||
logger.debug(
|
||||
f"Skipping {name} (model doesn't support guidance embeddings)"
|
||||
)
|
||||
continue
|
||||
|
||||
# Split fused qkv in double blocks into separate q/k/v projections
|
||||
match = re.match(
|
||||
r"double_blocks\.(\d+)\.(img_attn|txt_attn)\.qkv\.(weight|bias)$", name
|
||||
)
|
||||
if match:
|
||||
block_idx, attn_type, param_type = match.groups()
|
||||
hidden_size = qkv_size // 3
|
||||
|
||||
if tensor.shape[0] < 3 * hidden_size:
|
||||
logger.warning(
|
||||
f"{name} shape {tensor.shape} smaller than expected qkv size {3 * hidden_size}, skipping"
|
||||
)
|
||||
continue
|
||||
|
||||
if param_type == "bias":
|
||||
q_tensor = tensor[:hidden_size]
|
||||
k_tensor = tensor[hidden_size : 2 * hidden_size]
|
||||
v_tensor = tensor[2 * hidden_size : 3 * hidden_size]
|
||||
else:
|
||||
q_tensor = tensor[:hidden_size, :]
|
||||
k_tensor = tensor[hidden_size : 2 * hidden_size, :]
|
||||
v_tensor = tensor[2 * hidden_size : 3 * hidden_size, :]
|
||||
|
||||
target_prefix = f"transformer_blocks.{block_idx}.attn"
|
||||
if attn_type == "img_attn":
|
||||
yield f"{target_prefix}.to_q.{param_type}", q_tensor
|
||||
yield f"{target_prefix}.to_k.{param_type}", k_tensor
|
||||
yield f"{target_prefix}.to_v.{param_type}", v_tensor
|
||||
else:
|
||||
# txt_attn corresponds to encoder projections
|
||||
yield f"{target_prefix}.add_q_proj.{param_type}", q_tensor
|
||||
yield f"{target_prefix}.add_k_proj.{param_type}", k_tensor
|
||||
yield f"{target_prefix}.add_v_proj.{param_type}", v_tensor
|
||||
continue
|
||||
|
||||
match = re.match(r"single_blocks\.(\d+)\.linear1\.(weight|bias)$", name)
|
||||
if match:
|
||||
block_idx, param_type = match.groups()
|
||||
expected_size = qkv_size + mlp_hidden_dim
|
||||
|
||||
if tensor.shape[0] < expected_size:
|
||||
logger.warning(
|
||||
f"linear1.{param_type} shape {tensor.shape} doesn't match "
|
||||
f"expected size {expected_size}, skipping"
|
||||
)
|
||||
continue
|
||||
|
||||
# Split tensor
|
||||
qkv_tensor = (
|
||||
tensor[:qkv_size] if param_type == "bias" else tensor[:qkv_size, :]
|
||||
)
|
||||
mlp_tensor = (
|
||||
tensor[qkv_size:] if param_type == "bias" else tensor[qkv_size:, :]
|
||||
)
|
||||
|
||||
# Split qkv into q/k/v for single blocks
|
||||
hidden_size = qkv_size // 3
|
||||
if param_type == "bias":
|
||||
q_tensor = qkv_tensor[:hidden_size]
|
||||
k_tensor = qkv_tensor[hidden_size : 2 * hidden_size]
|
||||
v_tensor = qkv_tensor[2 * hidden_size : 3 * hidden_size]
|
||||
else:
|
||||
q_tensor = qkv_tensor[:hidden_size, :]
|
||||
k_tensor = qkv_tensor[hidden_size : 2 * hidden_size, :]
|
||||
v_tensor = qkv_tensor[2 * hidden_size : 3 * hidden_size, :]
|
||||
|
||||
yield (
|
||||
f"single_transformer_blocks.{block_idx}.attn.to_q.{param_type}",
|
||||
q_tensor,
|
||||
)
|
||||
yield (
|
||||
f"single_transformer_blocks.{block_idx}.attn.to_k.{param_type}",
|
||||
k_tensor,
|
||||
)
|
||||
yield (
|
||||
f"single_transformer_blocks.{block_idx}.attn.to_v.{param_type}",
|
||||
v_tensor,
|
||||
)
|
||||
yield (
|
||||
f"single_transformer_blocks.{block_idx}.proj_mlp.{param_type}",
|
||||
mlp_tensor,
|
||||
)
|
||||
elif name == "final_layer.adaLN_modulation.1.weight":
|
||||
# ComfyUI: output order is [shift, scale]
|
||||
# AdaLayerNormContinuous: expects [scale, shift]
|
||||
# Need to swap the first half and second half of the weight matrix
|
||||
# Weight shape: (2 * hidden_size, hidden_size)
|
||||
# Split into two halves and swap them
|
||||
half_size = tensor.shape[0] // 2
|
||||
shift_weights = tensor[:half_size, :]
|
||||
scale_weights = tensor[half_size:, :]
|
||||
# Swap: put scale first, then shift
|
||||
swapped_tensor = torch.cat([scale_weights, shift_weights], dim=0)
|
||||
logger.info(
|
||||
f"Swapped scale/shift order for {name}: "
|
||||
f"shape {tensor.shape} -> {swapped_tensor.shape}"
|
||||
)
|
||||
yield name, swapped_tensor
|
||||
elif name == "final_layer.adaLN_modulation.1.bias":
|
||||
# Same swap for bias: (2 * hidden_size,)
|
||||
half_size = tensor.shape[0] // 2
|
||||
shift_bias = tensor[:half_size]
|
||||
scale_bias = tensor[half_size:]
|
||||
swapped_tensor = torch.cat([scale_bias, shift_bias], dim=0)
|
||||
logger.info(
|
||||
f"Swapped scale/shift order for {name}: "
|
||||
f"shape {tensor.shape} -> {swapped_tensor.shape}"
|
||||
)
|
||||
yield name, swapped_tensor
|
||||
else:
|
||||
# Other weights pass through (handled by param_names_mapping)
|
||||
yield name, tensor
|
||||
|
||||
def _load_transformer_from_safetensors(
|
||||
self,
|
||||
server_args: ServerArgs,
|
||||
loaded_modules: dict[str, torch.nn.Module] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Load transformer directly from safetensors file without model_index.json.
|
||||
"""
|
||||
if loaded_modules is not None and "transformer" in loaded_modules:
|
||||
logger.info("Using provided transformer module")
|
||||
components = {
|
||||
"transformer": loaded_modules["transformer"],
|
||||
"scheduler": self.modules.get("scheduler"),
|
||||
}
|
||||
return components
|
||||
|
||||
if hasattr(server_args.pipeline_config, "dit_config"):
|
||||
dit_config = server_args.pipeline_config.dit_config
|
||||
if not isinstance(dit_config, FluxConfig):
|
||||
logger.warning("dit_config is not FluxConfig, creating new FluxConfig")
|
||||
dit_config = FluxConfig()
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
else:
|
||||
logger.info("Creating default FluxConfig")
|
||||
dit_config = FluxConfig()
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
|
||||
# Set guidance_embeds to True for ComfyUI Flux models
|
||||
dit_config.arch_config.guidance_embeds = True
|
||||
logger.info("Set guidance_embeds=True for ComfyUI Flux model")
|
||||
|
||||
if dit_config.arch_config.param_names_mapping is None:
|
||||
dit_config.arch_config.param_names_mapping = {}
|
||||
|
||||
# ComfyUI Flux uses different parameter names than SGLang Flux
|
||||
# Key differences:
|
||||
# - ComfyUI: single_blocks.{i}.linear1 (fused QKV + MLP input)
|
||||
# - SGLang: single_transformer_blocks.{i}.attn.to_qkv + proj_mlp (separate)
|
||||
# - ComfyUI: single_blocks.{i}.linear2
|
||||
# - SGLang: single_transformer_blocks.{i}.proj_out
|
||||
# - ComfyUI: double_blocks.{i}.img_attn.qkv / txt_attn.qkv
|
||||
# - SGLang: transformer_blocks.{i}.attn.to_qkv / attn.to_added_qkv
|
||||
|
||||
# Note: For fused layers like linear1, we need custom weight splitting logic
|
||||
# which will be handled in the weight conversion function below
|
||||
comfyui_flux_mappings = {
|
||||
# Double stream blocks - attention layers
|
||||
r"double_blocks\.(\d+)\.img_attn\.qkv\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.attn.to_qkv.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_attn\.qkv\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.attn.to_added_qkv.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_attn\.proj\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.attn.to_out.0.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_attn\.proj\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.attn.to_add_out.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_attn\.norm\.query_norm\.scale$": (
|
||||
r"transformer_blocks.\1.attn.norm_q.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_attn\.norm\.key_norm\.scale$": (
|
||||
r"transformer_blocks.\1.attn.norm_k.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_attn\.norm\.query_norm\.scale$": (
|
||||
r"transformer_blocks.\1.attn.norm_added_q.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_attn\.norm\.key_norm\.scale$": (
|
||||
r"transformer_blocks.\1.attn.norm_added_k.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
# Double stream blocks - MLP layers (map to net structure)
|
||||
r"double_blocks\.(\d+)\.img_mlp\.0\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.ff.net.0.proj.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.img_mlp\.2\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.ff.net.2.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_mlp\.0\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.ff_context.net.0.proj.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_mlp\.2\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.ff_context.net.2.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
# Double stream blocks - modulation layers
|
||||
r"double_blocks\.(\d+)\.img_mod\.lin\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.norm1.linear.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"double_blocks\.(\d+)\.txt_mod\.lin\.(weight|bias)$": (
|
||||
r"transformer_blocks.\1.norm1_context.linear.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
# Single stream blocks - linear2 maps to proj_out
|
||||
r"single_blocks\.(\d+)\.linear2\.(weight|bias)$": (
|
||||
r"single_transformer_blocks.\1.proj_out.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
# Single stream blocks - norm layers (scale -> weight)
|
||||
r"single_blocks\.(\d+)\.norm\.query_norm\.scale$": (
|
||||
r"single_transformer_blocks.\1.attn.norm_q.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"single_blocks\.(\d+)\.norm\.key_norm\.scale$": (
|
||||
r"single_transformer_blocks.\1.attn.norm_k.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
# Single stream blocks - modulation (maps to norm.linear)
|
||||
r"single_blocks\.(\d+)\.modulation\.lin\.(weight|bias)$": (
|
||||
r"single_transformer_blocks.\1.norm.linear.\2",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
# Time and guidance embeddings
|
||||
r"^time_in\.in_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.timestep_embedder.linear_1.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^time_in\.out_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.timestep_embedder.linear_2.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^txt_in\.(weight|bias)$": (r"context_embedder.\1", None, None),
|
||||
r"^vector_in\.in_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.text_embedder.linear_1.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^vector_in\.out_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.text_embedder.linear_2.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
# Final layer mappings
|
||||
r"^final_layer\.linear\.(weight|bias)$": (r"proj_out.\1", None, None),
|
||||
r"^final_layer\.norm_final\.(weight|bias)$": (r"norm_out.\1", None, None),
|
||||
r"^final_layer\.adaLN_modulation\.1\.(weight|bias)$": (
|
||||
r"norm_out.linear.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
# Image input embedding
|
||||
r"^img_in\.(weight|bias)$": (r"x_embedder.\1", None, None),
|
||||
# Guidance embeddings (if model supports guidance)
|
||||
r"^guidance_in\.in_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.guidance_embedder.linear_1.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^guidance_in\.out_layer\.(weight|bias)$": (
|
||||
r"time_text_embed.guidance_embedder.linear_2.\1",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
}
|
||||
|
||||
# Merge ComfyUI mappings with existing mappings (ComfyUI mappings take precedence)
|
||||
updated_mapping = {
|
||||
**dit_config.arch_config.param_names_mapping,
|
||||
**comfyui_flux_mappings,
|
||||
}
|
||||
dit_config.arch_config.param_names_mapping = updated_mapping
|
||||
logger.info(
|
||||
"Added ComfyUI weight name mappings for Flux model. "
|
||||
f"Total mappings: {len(updated_mapping)}"
|
||||
)
|
||||
|
||||
cls_name = "FluxTransformer2DModel"
|
||||
model_cls, _ = ModelRegistry.resolve_model_cls(cls_name)
|
||||
logger.info("Resolved transformer class: %s", cls_name)
|
||||
|
||||
original_mapping = None
|
||||
if comfyui_flux_mappings:
|
||||
original_mapping = model_cls.param_names_mapping
|
||||
model_cls.param_names_mapping = updated_mapping
|
||||
logger.info(
|
||||
"Temporarily updated model class param_names_mapping with ComfyUI mappings. "
|
||||
f"Total mappings: {len(updated_mapping)}"
|
||||
)
|
||||
|
||||
safetensors_list = [self.model_path]
|
||||
logger.info("Loading weights from: %s", safetensors_list)
|
||||
default_dtype = resolve_precision(
|
||||
server_args, "dit", precision_attr="dit_precision"
|
||||
)
|
||||
server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "."
|
||||
hf_config = {}
|
||||
|
||||
hidden_size = (
|
||||
dit_config.arch_config.num_attention_heads
|
||||
* dit_config.arch_config.attention_head_dim
|
||||
)
|
||||
mlp_ratio = getattr(dit_config.arch_config, "mlp_ratio", 4.0)
|
||||
mlp_hidden_dim = int(hidden_size * mlp_ratio)
|
||||
qkv_size = 3 * hidden_size
|
||||
has_guidance_embeds = True
|
||||
|
||||
# Load and convert weights from safetensors file
|
||||
model, reverse_param_names_mapping = (
|
||||
self._load_and_convert_weights_from_safetensors(
|
||||
model_cls=model_cls,
|
||||
dit_config=dit_config,
|
||||
hf_config=hf_config,
|
||||
safetensors_list=safetensors_list,
|
||||
updated_mapping=updated_mapping,
|
||||
qkv_size=qkv_size,
|
||||
mlp_hidden_dim=mlp_hidden_dim,
|
||||
has_guidance_embeds=has_guidance_embeds,
|
||||
default_dtype=default_dtype,
|
||||
)
|
||||
)
|
||||
|
||||
model = model.eval()
|
||||
for param in model.parameters():
|
||||
param.requires_grad = False
|
||||
|
||||
model.reverse_param_names_mapping = reverse_param_names_mapping
|
||||
|
||||
if original_mapping is not None:
|
||||
model_cls.param_names_mapping = original_mapping
|
||||
|
||||
total_params = sum(p.numel() for p in model.parameters())
|
||||
logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9)
|
||||
|
||||
components = {
|
||||
"transformer": model,
|
||||
"scheduler": self.modules.get("scheduler"),
|
||||
}
|
||||
|
||||
logger.info("Successfully loaded modules: %s", list(components.keys()))
|
||||
return components
|
||||
|
||||
def create_pipeline_stages(self, server_args: ServerArgs):
|
||||
logger.info(
|
||||
"ComfyUIFluxPipeline.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage"
|
||||
)
|
||||
|
||||
self.add_stages(
|
||||
[
|
||||
ComfyUILatentPreparationStage(
|
||||
scheduler=self.get_module("scheduler"),
|
||||
transformer=self.get_module("transformer"),
|
||||
),
|
||||
DenoisingStage(
|
||||
transformer=self.get_module("transformer"),
|
||||
scheduler=self.get_module("scheduler"),
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"ComfyUIFluxPipeline stages created: {list(self._stage_name_mapping.keys())}"
|
||||
)
|
||||
|
||||
|
||||
EntryClass = ComfyUIFluxPipeline
|
||||
@@ -1,359 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import os
|
||||
from itertools import chain
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
from torch.distributed import init_device_mesh
|
||||
from torch.distributed.fsdp import MixedPrecisionPolicy
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.qwenimage import QwenImageDitConfig
|
||||
from sglang.multimodal_gen.runtime.distributed import get_local_torch_device
|
||||
from sglang.multimodal_gen.runtime.loader.fsdp_load import (
|
||||
load_model_from_full_model_state_dict,
|
||||
shard_model,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.utils import (
|
||||
get_param_names_mapping,
|
||||
set_default_torch_dtype,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.weight_utils import (
|
||||
safetensors_weights_iterator,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
|
||||
from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import (
|
||||
ComfyUIPassThroughScheduler,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
ComposedPipelineBase,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages import (
|
||||
ComfyUILatentPreparationStage,
|
||||
DenoisingStage,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
|
||||
from sglang.multimodal_gen.runtime.utils.precision import (
|
||||
resolve_precision,
|
||||
set_mixed_precision_policy,
|
||||
)
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
class ComfyUIQwenImagePipelineBase(LoRAPipeline, ComposedPipelineBase):
|
||||
"""
|
||||
Base pipeline for ComfyUI QwenImage integration with only denoising stage.
|
||||
|
||||
This pipeline requires pre-processed inputs:
|
||||
- prompt_embeds: Pre-encoded text embeddings (list of tensors)
|
||||
- latents: Pre-processed image latents in sequence format [B, S, D]
|
||||
|
||||
Usage:
|
||||
generator = DiffGenerator.from_pretrained(
|
||||
model_path="path/to/model",
|
||||
pipeline_class_name="ComfyUIQwenImagePipeline",
|
||||
device="cuda",
|
||||
)
|
||||
"""
|
||||
|
||||
# Subclasses should override this
|
||||
zero_cond_t: bool = False
|
||||
|
||||
pipeline_name = "ComfyUIQwenImagePipeline"
|
||||
|
||||
_required_config_modules = [
|
||||
"transformer",
|
||||
"scheduler",
|
||||
]
|
||||
|
||||
def initialize_pipeline(self, server_args: ServerArgs):
|
||||
"""
|
||||
Initialize the pipeline with ComfyUI pass-through scheduler.
|
||||
This scheduler does not modify latents, allowing ComfyUI to handle denoising.
|
||||
"""
|
||||
self.modules["scheduler"] = ComfyUIPassThroughScheduler(
|
||||
num_train_timesteps=1000
|
||||
)
|
||||
|
||||
# Ensure VAE config is properly initialized even though we don't load the VAE model
|
||||
vae_config = server_args.pipeline_config.vae_config
|
||||
vae_config.post_init()
|
||||
logger.info(
|
||||
"Called vae_config.post_init() to set vae_scale_factor. "
|
||||
f"vae_scale_factor={vae_config.arch_config.vae_scale_factor}"
|
||||
)
|
||||
|
||||
def load_modules(
|
||||
self,
|
||||
server_args: ServerArgs,
|
||||
loaded_modules: dict[str, torch.nn.Module] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Load modules for ComfyUIQwenImagePipeline.
|
||||
|
||||
If model_path is a safetensors file, load transformer directly from it
|
||||
without requiring model_index.json. Otherwise, fall back to default loading.
|
||||
"""
|
||||
if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"):
|
||||
logger.info(
|
||||
"Detected safetensors file, loading transformer directly from: %s",
|
||||
self.model_path,
|
||||
)
|
||||
return self._load_transformer_from_safetensors(server_args, loaded_modules)
|
||||
else:
|
||||
logger.info(
|
||||
"Model path is a directory, using default loading method: %s",
|
||||
self.model_path,
|
||||
)
|
||||
return super().load_modules(server_args, loaded_modules)
|
||||
|
||||
def _load_transformer_from_safetensors(
|
||||
self,
|
||||
server_args: ServerArgs,
|
||||
loaded_modules: dict[str, torch.nn.Module] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Load transformer directly from safetensors without model_index.json."""
|
||||
|
||||
# 1) Fast path: use provided module
|
||||
if loaded_modules is not None and "transformer" in loaded_modules:
|
||||
logger.info("Using provided transformer module")
|
||||
return {
|
||||
"transformer": loaded_modules["transformer"],
|
||||
"scheduler": self.modules.get("scheduler"),
|
||||
}
|
||||
|
||||
# 2) Build config and mappings
|
||||
dit_config, updated_mapping, model_cls, default_dtype = (
|
||||
self._prepare_dit_config_and_mapping(server_args)
|
||||
)
|
||||
safetensors_list = [self.model_path]
|
||||
logger.info("Loading weights from: %s", safetensors_list)
|
||||
|
||||
# 3) Instantiate model (meta) and optionally shard
|
||||
model = self._instantiate_model(
|
||||
model_cls, dit_config, default_dtype, updated_mapping, server_args
|
||||
)
|
||||
|
||||
# 4) Load weights
|
||||
self._load_weights_into_model(
|
||||
model, safetensors_list, default_dtype, updated_mapping, server_args
|
||||
)
|
||||
|
||||
components = {
|
||||
"transformer": model,
|
||||
"scheduler": self.modules.get("scheduler"),
|
||||
}
|
||||
logger.info("Successfully loaded modules: %s", list(components.keys()))
|
||||
return components
|
||||
|
||||
def _prepare_dit_config_and_mapping(self, server_args: ServerArgs):
|
||||
from sglang.multimodal_gen.configs.models.dits.qwenimage import (
|
||||
QwenImageArchConfig,
|
||||
)
|
||||
|
||||
comfyui_arch_config = QwenImageArchConfig(
|
||||
patch_size=2,
|
||||
in_channels=64,
|
||||
out_channels=16,
|
||||
num_layers=60,
|
||||
attention_head_dim=128,
|
||||
num_attention_heads=24,
|
||||
joint_attention_dim=3584,
|
||||
pooled_projection_dim=768,
|
||||
guidance_embeds=False,
|
||||
axes_dims_rope=(16, 56, 56),
|
||||
zero_cond_t=self.zero_cond_t,
|
||||
)
|
||||
dit_config = QwenImageDitConfig(arch_config=comfyui_arch_config)
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
|
||||
if dit_config.arch_config.param_names_mapping is None:
|
||||
dit_config.arch_config.param_names_mapping = {}
|
||||
|
||||
comfyui_qwen_mappings = {r"^model\.diffusion_model\.(.*)$": r"\1"}
|
||||
updated_mapping = {
|
||||
**dit_config.arch_config.param_names_mapping,
|
||||
**comfyui_qwen_mappings,
|
||||
}
|
||||
dit_config.arch_config.param_names_mapping = updated_mapping
|
||||
logger.info(
|
||||
"Added ComfyUI weight name mappings to param_names_mapping. "
|
||||
f"Total mappings: {len(updated_mapping)}"
|
||||
)
|
||||
|
||||
cls_name = "QwenImageTransformer2DModel"
|
||||
model_cls, _ = ModelRegistry.resolve_model_cls(cls_name)
|
||||
logger.info("Resolved transformer class: %s", cls_name)
|
||||
|
||||
default_dtype = resolve_precision(
|
||||
server_args, "dit", precision_attr="dit_precision"
|
||||
)
|
||||
server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "."
|
||||
assert server_args.hsdp_shard_dim is not None, "hsdp_shard_dim must be set"
|
||||
logger.info(
|
||||
"Loading %s from safetensors file, default_dtype: %s",
|
||||
cls_name,
|
||||
default_dtype,
|
||||
)
|
||||
return dit_config, updated_mapping, model_cls, default_dtype
|
||||
|
||||
def _instantiate_model(
|
||||
self,
|
||||
model_cls,
|
||||
dit_config,
|
||||
default_dtype,
|
||||
updated_mapping,
|
||||
server_args: ServerArgs,
|
||||
):
|
||||
from sglang.multimodal_gen.runtime.platforms import current_platform
|
||||
|
||||
hf_config = {}
|
||||
original_mapping = model_cls.param_names_mapping
|
||||
model_cls.param_names_mapping = updated_mapping
|
||||
logger.info(
|
||||
"Temporarily updated model class param_names_mapping with ComfyUI mappings. "
|
||||
f"Total mappings: {len(updated_mapping)}"
|
||||
)
|
||||
|
||||
try:
|
||||
# precision-constraint: FSDP mixed precision currently uses bf16
|
||||
# parameters and fp32 reduction regardless of model load dtype.
|
||||
mp_policy = MixedPrecisionPolicy(
|
||||
torch.bfloat16, torch.float32, None, cast_forward_inputs=False
|
||||
)
|
||||
set_mixed_precision_policy(
|
||||
param_dtype=torch.bfloat16,
|
||||
reduce_dtype=torch.float32,
|
||||
output_dtype=None,
|
||||
mp_policy=mp_policy,
|
||||
)
|
||||
|
||||
with set_default_torch_dtype(default_dtype), torch.device("meta"):
|
||||
model = model_cls(**{"config": dit_config, "hf_config": hf_config})
|
||||
|
||||
use_fsdp = server_args.should_use_fsdp_for_component("transformer")
|
||||
if current_platform.is_mps():
|
||||
use_fsdp = False
|
||||
logger.info("Disabling FSDP for MPS platform as it's not compatible")
|
||||
|
||||
if use_fsdp:
|
||||
device_mesh = init_device_mesh(
|
||||
current_platform.device_type,
|
||||
mesh_shape=(
|
||||
server_args.hsdp_replicate_dim,
|
||||
server_args.hsdp_shard_dim,
|
||||
),
|
||||
mesh_dim_names=("replicate", "shard"),
|
||||
)
|
||||
shard_model(
|
||||
model,
|
||||
cpu_offload=False,
|
||||
reshard_after_forward=True,
|
||||
mp_policy=mp_policy,
|
||||
mesh=device_mesh,
|
||||
fsdp_shard_conditions=getattr(
|
||||
model, "_fsdp_shard_conditions", None
|
||||
),
|
||||
pin_cpu_memory=server_args.pin_cpu_memory,
|
||||
)
|
||||
finally:
|
||||
model_cls.param_names_mapping = original_mapping
|
||||
|
||||
return model
|
||||
|
||||
def _load_weights_into_model(
|
||||
self,
|
||||
model,
|
||||
safetensors_list,
|
||||
default_dtype,
|
||||
updated_mapping,
|
||||
server_args: ServerArgs,
|
||||
):
|
||||
use_fsdp = server_args.should_use_fsdp_for_component("transformer")
|
||||
component_starts_on_cpu = server_args.should_start_component_on_cpu(
|
||||
"transformer"
|
||||
)
|
||||
# Create weight iterator for loading
|
||||
weight_iterator = safetensors_weights_iterator(safetensors_list)
|
||||
|
||||
# Load weights
|
||||
param_names_mapping_fn = get_param_names_mapping(updated_mapping)
|
||||
load_model_from_full_model_state_dict(
|
||||
model,
|
||||
weight_iterator,
|
||||
get_local_torch_device(),
|
||||
default_dtype,
|
||||
strict=True,
|
||||
cpu_offload=component_starts_on_cpu and not use_fsdp,
|
||||
param_names_mapping=param_names_mapping_fn,
|
||||
)
|
||||
|
||||
# Check for meta parameters
|
||||
for n, p in chain(model.named_parameters(), model.named_buffers()):
|
||||
if p.is_meta:
|
||||
raise RuntimeError(f"Unexpected param or buffer {n} on meta device.")
|
||||
if isinstance(p, torch.nn.Parameter):
|
||||
p.requires_grad = False
|
||||
|
||||
total_params = sum(p.numel() for p in model.parameters())
|
||||
logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9)
|
||||
|
||||
def create_pipeline_stages(self, server_args: ServerArgs):
|
||||
logger.info(
|
||||
f"{self.__class__.__name__}.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage"
|
||||
)
|
||||
|
||||
self.add_stages(
|
||||
[
|
||||
ComfyUILatentPreparationStage(
|
||||
scheduler=self.get_module("scheduler"),
|
||||
transformer=self.get_module("transformer"),
|
||||
),
|
||||
DenoisingStage(
|
||||
transformer=self.get_module("transformer"),
|
||||
scheduler=self.get_module("scheduler"),
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"{self.__class__.__name__} stages created: {list(self._stage_name_mapping.keys())}"
|
||||
)
|
||||
|
||||
|
||||
class ComfyUIQwenImagePipeline(ComfyUIQwenImagePipelineBase):
|
||||
"""ComfyUI QwenImage pipeline for text-to-image generation."""
|
||||
|
||||
pipeline_name = "ComfyUIQwenImagePipeline"
|
||||
zero_cond_t = False
|
||||
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import (
|
||||
QwenImagePipelineConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.sample.qwenimage import QwenImageSamplingParams
|
||||
|
||||
pipeline_config_cls = QwenImagePipelineConfig
|
||||
sampling_params_cls = QwenImageSamplingParams
|
||||
|
||||
|
||||
class ComfyUIQwenImageEditPipeline(ComfyUIQwenImagePipelineBase):
|
||||
"""ComfyUI QwenImage pipeline for image-to-image editing."""
|
||||
|
||||
pipeline_name = "ComfyUIQwenImageEditPipeline"
|
||||
zero_cond_t = True
|
||||
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import (
|
||||
QwenImageEditPlusPipelineConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.sample.qwenimage import (
|
||||
QwenImageEditPlusSamplingParams,
|
||||
)
|
||||
|
||||
pipeline_config_cls = QwenImageEditPlusPipelineConfig
|
||||
sampling_params_cls = QwenImageEditPlusSamplingParams
|
||||
|
||||
|
||||
EntryClass = [ComfyUIQwenImagePipeline, ComfyUIQwenImageEditPipeline]
|
||||
@@ -1,412 +0,0 @@
|
||||
# Copied and adapted from: https://github.com/hao-ai-lab/FastVideo
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Generator
|
||||
from itertools import chain
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
from torch.distributed import init_device_mesh
|
||||
from torch.distributed.fsdp import MixedPrecisionPolicy
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.zimage import ZImageDitConfig
|
||||
from sglang.multimodal_gen.runtime.distributed import get_local_torch_device
|
||||
from sglang.multimodal_gen.runtime.loader.fsdp_load import (
|
||||
load_model_from_full_model_state_dict,
|
||||
shard_model,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.utils import (
|
||||
get_param_names_mapping,
|
||||
set_default_torch_dtype,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.weight_utils import (
|
||||
safetensors_weights_iterator,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
|
||||
from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import (
|
||||
ComfyUIPassThroughScheduler,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
ComposedPipelineBase,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages import (
|
||||
ComfyUILatentPreparationStage,
|
||||
DenoisingStage,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
|
||||
from sglang.multimodal_gen.runtime.utils.precision import (
|
||||
resolve_precision,
|
||||
set_mixed_precision_policy,
|
||||
)
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
class ComfyUIZImagePipeline(LoRAPipeline, ComposedPipelineBase):
|
||||
"""
|
||||
Simplified pipeline for ComfyUI integration with only denoising stage.
|
||||
|
||||
This pipeline requires pre-processed inputs:
|
||||
- prompt_embeds: Pre-encoded text embeddings (list of tensors)
|
||||
- negative_prompt_embeds: Pre-encoded negative prompt embeddings (if using CFG)
|
||||
- latents: Optional initial noise latents (will be generated if not provided)
|
||||
|
||||
Usage:
|
||||
generator = DiffGenerator.from_pretrained(
|
||||
model_path="path/to/model",
|
||||
pipeline_class_name="ComfyUIZImagePipeline",
|
||||
device="cuda",
|
||||
)
|
||||
"""
|
||||
|
||||
pipeline_name = "ComfyUIZImagePipeline"
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.zimage import (
|
||||
ZImagePipelineConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.sample.zimage import ZImageSamplingParams
|
||||
|
||||
pipeline_config_cls = ZImagePipelineConfig
|
||||
sampling_params_cls = ZImageSamplingParams
|
||||
|
||||
_required_config_modules = [
|
||||
"transformer",
|
||||
"scheduler",
|
||||
]
|
||||
|
||||
def initialize_pipeline(self, server_args: ServerArgs):
|
||||
"""
|
||||
Initialize the pipeline with ComfyUI pass-through scheduler.
|
||||
This scheduler does not modify latents, allowing ComfyUI to handle denoising.
|
||||
"""
|
||||
self.modules["scheduler"] = ComfyUIPassThroughScheduler(
|
||||
num_train_timesteps=1000
|
||||
)
|
||||
|
||||
# Ensure VAE config is properly initialized even though we don't load the VAE model
|
||||
# This is necessary because get_freqs_cis uses spatial_compression_ratio
|
||||
if hasattr(server_args.pipeline_config, "vae_config"):
|
||||
vae_config = server_args.pipeline_config.vae_config
|
||||
if hasattr(vae_config, "post_init") and not hasattr(
|
||||
vae_config, "_post_init_called"
|
||||
):
|
||||
vae_config.post_init()
|
||||
logger.info(
|
||||
"Called vae_config.post_init() to set spatial_compression_ratio. "
|
||||
f"spatial_compression_ratio={vae_config.arch_config.spatial_compression_ratio}"
|
||||
)
|
||||
|
||||
def load_modules(
|
||||
self,
|
||||
server_args: ServerArgs,
|
||||
loaded_modules: dict[str, torch.nn.Module] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Load modules for ComfyUIZImagePipeline.
|
||||
|
||||
If model_path is a safetensors file, load transformer directly from it
|
||||
without requiring model_index.json. Otherwise, fall back to default loading.
|
||||
"""
|
||||
if os.path.isfile(self.model_path) and self.model_path.endswith(".safetensors"):
|
||||
logger.info(
|
||||
"Detected safetensors file, loading transformer directly from: %s",
|
||||
self.model_path,
|
||||
)
|
||||
return self._load_transformer_from_safetensors(server_args, loaded_modules)
|
||||
else:
|
||||
logger.info(
|
||||
"Model path is a directory, using default loading method: %s",
|
||||
self.model_path,
|
||||
)
|
||||
return super().load_modules(server_args, loaded_modules)
|
||||
|
||||
def _convert_comfyui_qkv_weights(
|
||||
self,
|
||||
weight_iterator: Generator[tuple[str, torch.Tensor], None, None],
|
||||
dim: int,
|
||||
num_heads: int,
|
||||
num_kv_heads: int,
|
||||
) -> Generator[tuple[str, torch.Tensor], None, None]:
|
||||
"""
|
||||
Convert ComfyUI zimage qkv weights to SGLang format.
|
||||
Splits merged qkv.weight into separate to_q, to_k, to_v weights.
|
||||
|
||||
Args:
|
||||
weight_iterator: Iterator yielding (name, tensor) pairs from safetensors
|
||||
dim: Model dimension
|
||||
num_heads: Number of attention heads
|
||||
num_kv_heads: Number of key-value heads
|
||||
|
||||
Yields:
|
||||
(name, tensor) pairs with qkv weights split into to_q, to_k, to_v
|
||||
"""
|
||||
head_dim = dim // num_heads
|
||||
q_size = dim
|
||||
k_size = head_dim * num_kv_heads
|
||||
|
||||
for name, tensor in weight_iterator:
|
||||
# Match qkv weights in layers, noise_refiner, or context_refiner
|
||||
# Pattern: (layers|noise_refiner|context_refiner).{i}.attention.qkv.(weight|bias)
|
||||
match = re.match(
|
||||
r"(layers|noise_refiner|context_refiner)\.(\d+)\.attention\.qkv\.(weight|bias)$",
|
||||
name,
|
||||
)
|
||||
if match:
|
||||
module_name, layer_idx, param_type = match.groups()
|
||||
base_name = f"{module_name}.{layer_idx}.attention"
|
||||
|
||||
if param_type == "weight":
|
||||
# Weight shape: (q_size + k_size + v_size, dim)
|
||||
# Split into q, k, v
|
||||
q_weight = tensor[:q_size, :]
|
||||
k_weight = tensor[q_size : q_size + k_size, :]
|
||||
v_weight = tensor[q_size + k_size :, :]
|
||||
|
||||
logger.debug(
|
||||
f"Splitting {name} (shape {tensor.shape}) into "
|
||||
f"to_q ({q_weight.shape}), to_k ({k_weight.shape}), to_v ({v_weight.shape})"
|
||||
)
|
||||
|
||||
yield f"{base_name}.to_q.weight", q_weight
|
||||
yield f"{base_name}.to_k.weight", k_weight
|
||||
yield f"{base_name}.to_v.weight", v_weight
|
||||
else: # bias
|
||||
# Bias shape: (q_size + k_size + v_size,)
|
||||
# Split into q, k, v
|
||||
q_bias = tensor[:q_size]
|
||||
k_bias = tensor[q_size : q_size + k_size]
|
||||
v_bias = tensor[q_size + k_size :]
|
||||
|
||||
logger.debug(
|
||||
f"Splitting {name} (shape {tensor.shape}) into "
|
||||
f"to_q ({q_bias.shape}), to_k ({k_bias.shape}), to_v ({v_bias.shape})"
|
||||
)
|
||||
|
||||
yield f"{base_name}.to_q.bias", q_bias
|
||||
yield f"{base_name}.to_k.bias", k_bias
|
||||
yield f"{base_name}.to_v.bias", v_bias
|
||||
else:
|
||||
# Pass through other weights unchanged
|
||||
yield name, tensor
|
||||
|
||||
def _load_transformer_from_safetensors(
|
||||
self,
|
||||
server_args: ServerArgs,
|
||||
loaded_modules: dict[str, torch.nn.Module] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Load transformer directly from safetensors file without model_index.json.
|
||||
|
||||
This method:
|
||||
1. Uses hardcoded ZImageDitConfig for zimage model
|
||||
2. Loads transformer from the safetensors file
|
||||
3. Uses ComfyUIPassThroughScheduler (already created in initialize_pipeline)
|
||||
"""
|
||||
# Check if transformer is already provided
|
||||
if loaded_modules is not None and "transformer" in loaded_modules:
|
||||
logger.info("Using provided transformer module")
|
||||
components = {
|
||||
"transformer": loaded_modules["transformer"],
|
||||
"scheduler": self.modules.get("scheduler"),
|
||||
}
|
||||
return components
|
||||
|
||||
if hasattr(server_args.pipeline_config, "dit_config"):
|
||||
dit_config = server_args.pipeline_config.dit_config
|
||||
if not isinstance(dit_config, ZImageDitConfig):
|
||||
logger.warning(
|
||||
"dit_config is not ZImageDitConfig, creating new ZImageDitConfig"
|
||||
)
|
||||
dit_config = ZImageDitConfig()
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
else:
|
||||
logger.info("Creating default ZImageDitConfig")
|
||||
dit_config = ZImageDitConfig()
|
||||
server_args.pipeline_config.dit_config = dit_config
|
||||
|
||||
if dit_config.arch_config.param_names_mapping is None:
|
||||
dit_config.arch_config.param_names_mapping = {}
|
||||
|
||||
# Add mappings for norm layers: map from ComfyUI format (k_norm/q_norm) to SGLang format (norm_k/norm_q)
|
||||
# The regex matches the source name from safetensors, and the tuple specifies the target name in the model
|
||||
# Note: qkv weights are handled separately by _convert_comfyui_qkv_weights function
|
||||
comfyui_norm_mappings = {
|
||||
r"(.*)\.attention\.k_norm\.weight$": (
|
||||
r"\1.attention.norm_k.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"(.*)\.attention\.q_norm\.weight$": (
|
||||
r"\1.attention.norm_q.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"(.*)\.attention\.out\.weight$": (
|
||||
r"\1.attention.to_out.0.weight",
|
||||
None,
|
||||
None,
|
||||
),
|
||||
r"^final_layer\.(.*)$": (r"all_final_layer.2-1.\1", None, None),
|
||||
r"^x_embedder\.(.*)$": (r"all_x_embedder.2-1.\1", None, None),
|
||||
}
|
||||
|
||||
# Merge ComfyUI mappings with existing mappings (ComfyUI mappings take precedence)
|
||||
updated_mapping = {
|
||||
**dit_config.arch_config.param_names_mapping,
|
||||
**comfyui_norm_mappings,
|
||||
}
|
||||
dit_config.arch_config.param_names_mapping = updated_mapping
|
||||
logger.info(
|
||||
"Added ComfyUI weight name mappings (k_norm/q_norm -> norm_k/norm_q) to param_names_mapping. "
|
||||
f"Total mappings: {len(updated_mapping)}"
|
||||
)
|
||||
|
||||
cls_name = "ZImageTransformer2DModel"
|
||||
model_cls, _ = ModelRegistry.resolve_model_cls(cls_name)
|
||||
logger.info("Resolved transformer class: %s", cls_name)
|
||||
safetensors_list = [self.model_path]
|
||||
logger.info("Loading weights from: %s", safetensors_list)
|
||||
|
||||
default_dtype = resolve_precision(
|
||||
server_args, "dit", precision_attr="dit_precision"
|
||||
)
|
||||
server_args.model_paths["transformer"] = os.path.dirname(self.model_path) or "."
|
||||
hf_config = {}
|
||||
|
||||
assert server_args.hsdp_shard_dim is not None, "hsdp_shard_dim must be set"
|
||||
logger.info(
|
||||
"Loading %s from safetensors file, default_dtype: %s",
|
||||
cls_name,
|
||||
default_dtype,
|
||||
)
|
||||
|
||||
original_mapping = model_cls.param_names_mapping
|
||||
model_cls.param_names_mapping = updated_mapping
|
||||
logger.info(
|
||||
"Temporarily updated model class param_names_mapping with ComfyUI mappings. "
|
||||
f"Total mappings: {len(updated_mapping)}"
|
||||
)
|
||||
|
||||
try:
|
||||
# Create model first (same as maybe_load_fsdp_model)
|
||||
from sglang.multimodal_gen.runtime.platforms import current_platform
|
||||
|
||||
# precision-constraint: FSDP mixed precision currently uses bf16
|
||||
# parameters and fp32 reduction regardless of model load dtype.
|
||||
mp_policy = MixedPrecisionPolicy(
|
||||
torch.bfloat16, torch.float32, None, cast_forward_inputs=False
|
||||
)
|
||||
|
||||
set_mixed_precision_policy(
|
||||
param_dtype=torch.bfloat16,
|
||||
reduce_dtype=torch.float32,
|
||||
output_dtype=None,
|
||||
mp_policy=mp_policy,
|
||||
)
|
||||
|
||||
with set_default_torch_dtype(default_dtype), torch.device("meta"):
|
||||
model = model_cls(**{"config": dit_config, "hf_config": hf_config})
|
||||
|
||||
# Check if we should use FSDP
|
||||
use_fsdp = server_args.should_use_fsdp_for_component("transformer")
|
||||
component_starts_on_cpu = server_args.should_start_component_on_cpu(
|
||||
"transformer"
|
||||
)
|
||||
if current_platform.is_mps():
|
||||
use_fsdp = False
|
||||
logger.info("Disabling FSDP for MPS platform as it's not compatible")
|
||||
|
||||
if use_fsdp:
|
||||
device_mesh = init_device_mesh(
|
||||
current_platform.device_type,
|
||||
mesh_shape=(
|
||||
server_args.hsdp_replicate_dim,
|
||||
server_args.hsdp_shard_dim,
|
||||
),
|
||||
mesh_dim_names=("replicate", "shard"),
|
||||
)
|
||||
shard_model(
|
||||
model,
|
||||
cpu_offload=False,
|
||||
reshard_after_forward=True,
|
||||
mp_policy=mp_policy,
|
||||
mesh=device_mesh,
|
||||
fsdp_shard_conditions=getattr(
|
||||
model, "_fsdp_shard_conditions", None
|
||||
),
|
||||
pin_cpu_memory=server_args.pin_cpu_memory,
|
||||
)
|
||||
|
||||
# Get model dimensions for qkv splitting
|
||||
arch_config = dit_config.arch_config
|
||||
dim = arch_config.dim
|
||||
num_heads = arch_config.num_attention_heads
|
||||
num_kv_heads = arch_config.n_kv_heads
|
||||
|
||||
# Create weight iterator with qkv conversion
|
||||
base_weight_iterator = safetensors_weights_iterator(safetensors_list)
|
||||
converted_weight_iterator = self._convert_comfyui_qkv_weights(
|
||||
base_weight_iterator, dim, num_heads, num_kv_heads
|
||||
)
|
||||
|
||||
# Load weights
|
||||
param_names_mapping_fn = get_param_names_mapping(updated_mapping)
|
||||
load_model_from_full_model_state_dict(
|
||||
model,
|
||||
converted_weight_iterator,
|
||||
get_local_torch_device(),
|
||||
default_dtype,
|
||||
strict=True,
|
||||
cpu_offload=component_starts_on_cpu and not use_fsdp,
|
||||
param_names_mapping=param_names_mapping_fn,
|
||||
)
|
||||
|
||||
# Check for meta parameters
|
||||
for n, p in chain(model.named_parameters(), model.named_buffers()):
|
||||
if p.is_meta:
|
||||
raise RuntimeError(
|
||||
f"Unexpected param or buffer {n} on meta device."
|
||||
)
|
||||
if isinstance(p, torch.nn.Parameter):
|
||||
p.requires_grad = False
|
||||
finally:
|
||||
model_cls.param_names_mapping = original_mapping
|
||||
|
||||
total_params = sum(p.numel() for p in model.parameters())
|
||||
logger.info("Loaded transformer with %.2fB parameters", total_params / 1e9)
|
||||
|
||||
components = {
|
||||
"transformer": model,
|
||||
"scheduler": self.modules.get("scheduler"),
|
||||
}
|
||||
|
||||
logger.info("Successfully loaded modules: %s", list(components.keys()))
|
||||
return components
|
||||
|
||||
def create_pipeline_stages(self, server_args: ServerArgs):
|
||||
logger.info(
|
||||
"ComfyUIZImagePipeline.create_pipeline_stages() called - creating latent_preparation_stage and denoising_stage"
|
||||
)
|
||||
|
||||
self.add_stages(
|
||||
[
|
||||
ComfyUILatentPreparationStage(
|
||||
scheduler=self.get_module("scheduler"),
|
||||
transformer=self.get_module("transformer"),
|
||||
),
|
||||
DenoisingStage(
|
||||
transformer=self.get_module("transformer"),
|
||||
scheduler=self.get_module("scheduler"),
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"ComfyUIZImagePipeline stages created: {list(self._stage_name_mapping.keys())}"
|
||||
)
|
||||
|
||||
|
||||
EntryClass = ComfyUIZImagePipeline
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig
|
||||
from sglang.multimodal_gen.configs.sample.flux import FluxSamplingParams
|
||||
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
ComposedPipelineBase,
|
||||
@@ -32,6 +34,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs):
|
||||
class FluxPipeline(LoRAPipeline, ComposedPipelineBase):
|
||||
pipeline_name = "FluxPipeline"
|
||||
|
||||
# Used when the checkpoint is a single safetensors file with no
|
||||
# model_index.json to derive these from.
|
||||
pipeline_config_cls = FluxPipelineConfig
|
||||
sampling_params_cls = FluxSamplingParams
|
||||
|
||||
_required_config_modules = [
|
||||
"text_encoder",
|
||||
"text_encoder_2",
|
||||
|
||||
@@ -190,6 +190,15 @@ class MiniMaxH3Pipeline(LoRAPipeline, ComposedPipelineBase):
|
||||
)
|
||||
)
|
||||
|
||||
def create_comfyui_stages(self, server_args: ServerArgs) -> None:
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.comfyui_step import (
|
||||
MiniMaxH3ComfyUIStepStage,
|
||||
)
|
||||
|
||||
self.add_stage(
|
||||
MiniMaxH3ComfyUIStepStage(transformer=self.get_module("transformer"))
|
||||
)
|
||||
|
||||
|
||||
class FastH3Pipeline(MiniMaxH3Pipeline):
|
||||
"""FastH3: 4-step DMD2-distilled MiniMax-H3 (t2va only).
|
||||
|
||||
@@ -3,6 +3,14 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from diffusers.image_processor import VaeImageProcessor
|
||||
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import (
|
||||
QwenImageEditPlusPipelineConfig,
|
||||
QwenImagePipelineConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.sample.qwenimage import (
|
||||
QwenImageEditPlusSamplingParams,
|
||||
QwenImageSamplingParams,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.disaggregation.roles import RoleType
|
||||
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
@@ -39,6 +47,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs):
|
||||
class QwenImagePipeline(LoRAPipeline, ComposedPipelineBase):
|
||||
pipeline_name = "QwenImagePipeline"
|
||||
|
||||
# Used when the checkpoint is a single safetensors file with no
|
||||
# model_index.json to derive these from.
|
||||
pipeline_config_cls = QwenImagePipelineConfig
|
||||
sampling_params_cls = QwenImageSamplingParams
|
||||
|
||||
_required_config_modules = [
|
||||
"text_encoder",
|
||||
"tokenizer",
|
||||
@@ -84,6 +97,11 @@ class QwenImageEditPipeline(LoRAPipeline, ComposedPipelineBase):
|
||||
class QwenImageEditPlusPipeline(QwenImageEditPipeline):
|
||||
pipeline_name = "QwenImageEditPlusPipeline"
|
||||
|
||||
# Used when the checkpoint is a single safetensors file with no
|
||||
# model_index.json to derive these from.
|
||||
pipeline_config_cls = QwenImageEditPlusPipelineConfig
|
||||
sampling_params_cls = QwenImageEditPlusSamplingParams
|
||||
|
||||
|
||||
def prepare_mu_layered(batch: Req, server_args: ServerArgs):
|
||||
base_seqlen = 256 * 256 / 16 / 16
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# Copied and adapted from: https://github.com/hao-ai-lab/FastVideo
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.zimage import ZImagePipelineConfig
|
||||
from sglang.multimodal_gen.configs.sample.zimage import ZImageSamplingParams
|
||||
from sglang.multimodal_gen.runtime.pipelines_core import LoRAPipeline, Req
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
ComposedPipelineBase,
|
||||
@@ -27,6 +29,11 @@ def prepare_mu(batch: Req, server_args: ServerArgs):
|
||||
class ZImagePipeline(LoRAPipeline, ComposedPipelineBase):
|
||||
pipeline_name = "ZImagePipeline"
|
||||
|
||||
# Used when the checkpoint is a single safetensors file with no
|
||||
# model_index.json to derive these from.
|
||||
pipeline_config_cls = ZImagePipelineConfig
|
||||
sampling_params_cls = ZImageSamplingParams
|
||||
|
||||
_required_config_modules = [
|
||||
"text_encoder",
|
||||
"tokenizer",
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Worker-side support for ``--comfyui-mode``.
|
||||
|
||||
ComfyUI owns the sampler loop and calls SGLang once per DiT step. This
|
||||
module covers both halves of that contract:
|
||||
|
||||
- pipeline assembly: drop every module except the transformer, install a
|
||||
pass-through scheduler, keep only the stages needed for one forward
|
||||
- run cache: after the first step, text embeddings and packed extras stay
|
||||
on the worker; later steps send latents and the timestep
|
||||
|
||||
Single-file / GGUF DiT loading lives in
|
||||
``sglang.multimodal_gen.runtime.loader.comfyui_checkpoints``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any, Callable
|
||||
|
||||
from sglang.multimodal_gen.runtime.models.schedulers.scheduling_comfyui_passthrough import (
|
||||
ComfyUIPassThroughScheduler,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
ComposedPipelineBase,
|
||||
)
|
||||
|
||||
COMFYUI_REQUIRED_MODULES = ["transformer", "scheduler"]
|
||||
|
||||
_CONDITIONING_FIELDS = (
|
||||
"prompt_embeds",
|
||||
"negative_prompt_embeds",
|
||||
"prompt_seq_lens",
|
||||
"negative_prompt_seq_lens",
|
||||
"pooled_embeds",
|
||||
"neg_pooled_embeds",
|
||||
"image_latent",
|
||||
"vae_image_sizes",
|
||||
"prompt_attention_mask",
|
||||
"negative_attention_mask",
|
||||
"prompt_embeds_mask",
|
||||
"negative_prompt_embeds_mask",
|
||||
"sigmas",
|
||||
)
|
||||
_SESSION_SKIP_EXTRA = frozenset(
|
||||
{"comfyui_session_id", "comfyui_cond_key", "comfyui_cache_fp"}
|
||||
)
|
||||
_SESSIONS: dict[str, dict[str, Any]] = {}
|
||||
_RUNS: dict[str, Any] = {}
|
||||
|
||||
|
||||
def is_comfyui_mode(server_args: ServerArgs) -> bool:
|
||||
return bool(server_args.comfyui_mode)
|
||||
|
||||
|
||||
def initialize_comfyui_pipeline(
|
||||
pipeline: ComposedPipelineBase, server_args: ServerArgs
|
||||
) -> None:
|
||||
"""Install the pass-through scheduler and finish deriving VAE geometry.
|
||||
|
||||
The VAE model itself is never loaded, but its config still carries the
|
||||
compression ratios that RoPE frequency construction reads.
|
||||
"""
|
||||
pipeline.modules["scheduler"] = ComfyUIPassThroughScheduler(
|
||||
num_train_timesteps=1000
|
||||
)
|
||||
|
||||
vae_config = getattr(server_args.pipeline_config, "vae_config", None)
|
||||
if (
|
||||
vae_config is not None
|
||||
and hasattr(vae_config, "post_init")
|
||||
and not hasattr(vae_config, "_post_init_called")
|
||||
):
|
||||
# Flux / Qwen RoPE reads arch_config.vae_scale_factor, which
|
||||
# post_init() derives from dim_mult. Do not skip just because a
|
||||
# ComfyUI DiT checkpoint has no latents_mean — that field is only
|
||||
# required when this process owns the VAE (MiniMax-H3 native).
|
||||
vae_config.post_init()
|
||||
|
||||
|
||||
def create_comfyui_pipeline_stages(
|
||||
pipeline: ComposedPipelineBase, server_args: ServerArgs
|
||||
) -> None:
|
||||
if hasattr(pipeline, "create_comfyui_stages"):
|
||||
pipeline.create_comfyui_stages(server_args)
|
||||
return
|
||||
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages import (
|
||||
ComfyUILatentPreparationStage,
|
||||
DenoisingStage,
|
||||
)
|
||||
|
||||
transformer = pipeline.get_module("transformer")
|
||||
scheduler = pipeline.get_module("scheduler")
|
||||
pipeline.add_stages(
|
||||
[
|
||||
ComfyUILatentPreparationStage(scheduler=scheduler, transformer=transformer),
|
||||
DenoisingStage(transformer=transformer, scheduler=scheduler),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def session_id_from_req(req) -> str | None:
|
||||
extra = getattr(req, "extra", None) or {}
|
||||
sid = extra.get("comfyui_session_id")
|
||||
return sid if sid else None
|
||||
|
||||
|
||||
def _session_family(session_id: str) -> str:
|
||||
return session_id.split(":", 1)[0]
|
||||
|
||||
|
||||
def _evict_other_runs(session_id: str) -> None:
|
||||
"""Drop older ``{uuid}:{run}`` slots for this executor."""
|
||||
prefix = _session_family(session_id) + ":"
|
||||
for store in (_SESSIONS, _RUNS):
|
||||
for key in list(store):
|
||||
if key.startswith(prefix) and key != session_id:
|
||||
store.pop(key, None)
|
||||
|
||||
|
||||
def bind_comfyui_session(req):
|
||||
"""Restore cached conditioning, then refresh the cache from whatever is set.
|
||||
|
||||
Conditioning is stored per ``comfyui_cond_key`` so CFG pos/neg do not
|
||||
overwrite each other. Packed extras are restored only when
|
||||
``comfyui_cache_fp`` matches; a mismatch must not revive a stale layout.
|
||||
"""
|
||||
sid = session_id_from_req(req)
|
||||
if not sid:
|
||||
return req
|
||||
|
||||
_evict_other_runs(sid)
|
||||
extra = dict(getattr(req, "extra", None) or {})
|
||||
cond_key = extra.get("comfyui_cond_key") or "_"
|
||||
cached = _SESSIONS.get(sid) or {}
|
||||
cond_snap = (cached.get("_conds") or {}).get(cond_key) or {}
|
||||
current_fp = extra.get("comfyui_cache_fp")
|
||||
cached_fp = cached.get("_fp")
|
||||
extra_ok = current_fp is None or cached_fp is None or current_fp == cached_fp
|
||||
|
||||
for name, value in cond_snap.items():
|
||||
if _is_empty(getattr(req, name, None)):
|
||||
setattr(req, name, value)
|
||||
if extra_ok:
|
||||
for key, value in (cached.get("_extra") or {}).items():
|
||||
if _is_empty(extra.get(key)):
|
||||
extra[key] = value
|
||||
req.extra = extra
|
||||
|
||||
new_cond = {}
|
||||
for name in _CONDITIONING_FIELDS:
|
||||
value = getattr(req, name, None)
|
||||
if not _is_empty(value):
|
||||
new_cond[name] = value
|
||||
extra = getattr(req, "extra", None) or {}
|
||||
extra_snapshot = {}
|
||||
for key, value in extra.items():
|
||||
if key in _SESSION_SKIP_EXTRA or _is_empty(value):
|
||||
continue
|
||||
extra_snapshot[key] = value
|
||||
conds = dict(cached.get("_conds") or {})
|
||||
if new_cond:
|
||||
conds[cond_key] = new_cond
|
||||
snapshot: dict[str, Any] = {"_conds": conds}
|
||||
if extra_snapshot:
|
||||
snapshot["_extra"] = extra_snapshot
|
||||
if current_fp is not None:
|
||||
snapshot["_fp"] = current_fp
|
||||
elif extra_ok and cached_fp is not None:
|
||||
snapshot["_fp"] = cached_fp
|
||||
if conds or extra_snapshot:
|
||||
_SESSIONS[sid] = snapshot
|
||||
return req
|
||||
|
||||
|
||||
def get_run_state(req):
|
||||
sid = session_id_from_req(req)
|
||||
if sid:
|
||||
return _RUNS.get(sid)
|
||||
return None
|
||||
|
||||
|
||||
def set_run_state(req, state: Any) -> None:
|
||||
sid = session_id_from_req(req)
|
||||
if sid:
|
||||
_RUNS[sid] = state
|
||||
|
||||
|
||||
def pop_run_state(req):
|
||||
sid = session_id_from_req(req)
|
||||
if sid:
|
||||
return _RUNS.pop(sid, None)
|
||||
return None
|
||||
|
||||
|
||||
def get_or_create_run_state(req, factory: Callable[[], Any]):
|
||||
"""Session-scoped opaque state. Created once per run."""
|
||||
existing = get_run_state(req)
|
||||
if existing is not None:
|
||||
return existing
|
||||
state = factory()
|
||||
set_run_state(req, state)
|
||||
return state
|
||||
|
||||
|
||||
def release_run_state(session_id: str | None) -> None:
|
||||
"""Drop per-run objects; keep cached conditioning for the next step."""
|
||||
if session_id:
|
||||
_RUNS.pop(session_id, None)
|
||||
|
||||
|
||||
def release_comfyui_session(session_id: str | None) -> None:
|
||||
if session_id:
|
||||
_SESSIONS.pop(session_id, None)
|
||||
_RUNS.pop(session_id, None)
|
||||
|
||||
|
||||
def _is_empty(value) -> bool:
|
||||
if value is None:
|
||||
return True
|
||||
if isinstance(value, (list, tuple)):
|
||||
return len(value) == 0
|
||||
return False
|
||||
@@ -18,6 +18,10 @@ from sglang.multimodal_gen.runtime.disaggregation.roles import (
|
||||
RoleType,
|
||||
filter_modules_for_role,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import (
|
||||
is_comfyui_single_file,
|
||||
load_comfyui_transformer,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.component_loaders.component_loader import (
|
||||
ComponentLoader,
|
||||
PipelineComponentLoader,
|
||||
@@ -32,6 +36,12 @@ from sglang.multimodal_gen.runtime.managers.memory_managers.component_manager im
|
||||
ComponentResidencyStrategy,
|
||||
get_global_component_residency_manager,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import (
|
||||
COMFYUI_REQUIRED_MODULES,
|
||||
create_comfyui_pipeline_stages,
|
||||
initialize_comfyui_pipeline,
|
||||
is_comfyui_mode,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.executors.pipeline_executor import (
|
||||
PipelineExecutor,
|
||||
)
|
||||
@@ -123,6 +133,9 @@ class ComposedPipelineBase(ABC):
|
||||
)
|
||||
if base_required_config_modules is None:
|
||||
raise NotImplementedError("Subclass must set _required_config_modules")
|
||||
if is_comfyui_mode(server_args):
|
||||
# ComfyUI owns text encoding, VAE decode, and the sampler loop.
|
||||
base_required_config_modules = COMFYUI_REQUIRED_MODULES
|
||||
self._unfiltered_required_config_modules = tuple(base_required_config_modules)
|
||||
self._required_config_modules = list(self._unfiltered_required_config_modules)
|
||||
self._extra_config_module_map = dict(self._extra_config_module_map)
|
||||
@@ -166,6 +179,12 @@ class ComposedPipelineBase(ABC):
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
assert self.server_args is not None, "server_args must be set"
|
||||
if is_comfyui_mode(self.server_args):
|
||||
initialize_comfyui_pipeline(self, self.server_args)
|
||||
logger.info("Creating ComfyUI pipeline stages...")
|
||||
create_comfyui_pipeline_stages(self, self.server_args)
|
||||
return
|
||||
|
||||
self.initialize_pipeline(self.server_args)
|
||||
|
||||
logger.info("Creating pipeline stages...")
|
||||
@@ -425,6 +444,8 @@ class ComposedPipelineBase(ABC):
|
||||
loaded_modules: Optional[Dict[str, torch.nn.Module]] = None,
|
||||
If provided, loaded_modules will be used instead of loading from config/pretrained weights.
|
||||
"""
|
||||
if is_comfyui_mode(server_args) and is_comfyui_single_file(self.model_path):
|
||||
return load_comfyui_transformer(self, server_args, loaded_modules)
|
||||
|
||||
model_index = self._load_config()
|
||||
logger.info("Loading pipeline modules from config: %s", model_index)
|
||||
|
||||
@@ -155,7 +155,7 @@ class Req:
|
||||
raw_latent_shape: torch.Tensor | None = None
|
||||
did_sp_shard_latents: bool = False
|
||||
sp_video_start_frame: int = 0
|
||||
noise_pred: torch.Tensor | None = None
|
||||
noise_pred: torch.Tensor | list | tuple | None = None
|
||||
# vae-encoded condition image
|
||||
image_latent: torch.Tensor | list[torch.Tensor] | None = None
|
||||
condition_image_latent_ids: torch.Tensor | list[torch.Tensor] | None = None
|
||||
@@ -502,7 +502,7 @@ class OutputBatch:
|
||||
metrics_list: Optional[list[Optional[RequestMetrics]]] = None
|
||||
|
||||
# For ComfyUI integration: noise prediction from denoising stage
|
||||
noise_pred: torch.Tensor | None = None
|
||||
noise_pred: torch.Tensor | list | tuple | None = None
|
||||
peak_memory_mb: float = 0.0
|
||||
usage: dict[str, Any] | None = None
|
||||
|
||||
|
||||
+18
-90
@@ -1,114 +1,42 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""
|
||||
ComfyUI latent preparation stage with device mismatch fix.
|
||||
This stage extends LatentPreparationStage to handle device mismatch issues
|
||||
that occur when tensors are pickled and unpickled via broadcast_pyobj in
|
||||
multi-GPU scenarios.
|
||||
"""ComfyUI latent prep: restore the worker session and bind the pass-through scheduler.
|
||||
|
||||
Multi-rank hops move CUDA tensors with NCCL, so this stage no longer walks
|
||||
every field to fix pickle/gloo device mismatches.
|
||||
"""
|
||||
|
||||
import dataclasses
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.runtime.distributed import (
|
||||
get_local_torch_device,
|
||||
get_sp_group,
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import (
|
||||
bind_comfyui_session,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.diffusion_scheduler_utils import (
|
||||
get_or_create_request_scheduler,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.distributed.parallel_state import get_sp_world_size
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.latent_preparation import (
|
||||
LatentPreparationStage,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.utils.logging_utils import init_logger
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
class ComfyUILatentPreparationStage(LatentPreparationStage):
|
||||
"""
|
||||
ComfyUI-specific latent preparation stage with device mismatch fix.
|
||||
"""One DiT step: restore cached conditioning, then prepare latents."""
|
||||
|
||||
This stage extends LatentPreparationStage to automatically fix device
|
||||
mismatches for tensor fields on non-source ranks in multi-GPU scenarios.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _fix_tensor_device(value, target_device):
|
||||
"""Recursively fix tensor device, handling single tensors, lists, and tuples."""
|
||||
if isinstance(value, torch.Tensor):
|
||||
if value.device != target_device:
|
||||
return value.detach().clone().to(target_device)
|
||||
return value
|
||||
elif isinstance(value, list):
|
||||
return [
|
||||
ComfyUILatentPreparationStage._fix_tensor_device(v, target_device)
|
||||
for v in value
|
||||
]
|
||||
elif isinstance(value, tuple):
|
||||
return tuple(
|
||||
ComfyUILatentPreparationStage._fix_tensor_device(v, target_device)
|
||||
for v in value
|
||||
)
|
||||
return value
|
||||
|
||||
@staticmethod
|
||||
def _has_tensor(value):
|
||||
"""Check if value contains any tensor."""
|
||||
if isinstance(value, torch.Tensor):
|
||||
return True
|
||||
elif isinstance(value, (list, tuple)):
|
||||
return any(ComfyUILatentPreparationStage._has_tensor(v) for v in value)
|
||||
return False
|
||||
def verify_input(self, batch, server_args):
|
||||
bind_comfyui_session(batch)
|
||||
return super().verify_input(batch, server_args)
|
||||
|
||||
def forward(self, batch, server_args):
|
||||
"""
|
||||
Prepare latents with device mismatch fix for ComfyUI pipelines.
|
||||
|
||||
This method first fixes device mismatches for all tensor fields,
|
||||
then calls the parent class's forward method, and ensures raw_latent_shape
|
||||
is set correctly (before packing, for proper unpadding later).
|
||||
"""
|
||||
# Fix device mismatch for tensor fields on non-source ranks
|
||||
if get_sp_world_size() > 1:
|
||||
sp_group = get_sp_group()
|
||||
target_device = get_local_torch_device()
|
||||
|
||||
if sp_group.rank != 0:
|
||||
logger.debug(
|
||||
f"[ComfyUILatentPreparationStage] Fixing tensor device on rank={sp_group.rank} "
|
||||
f"target_device={target_device}"
|
||||
)
|
||||
|
||||
if dataclasses.is_dataclass(batch):
|
||||
for field in dataclasses.fields(batch):
|
||||
value = getattr(batch, field.name, None)
|
||||
if value is not None and self._has_tensor(value):
|
||||
fixed_value = self._fix_tensor_device(value, target_device)
|
||||
setattr(batch, field.name, fixed_value)
|
||||
else:
|
||||
for attr_name in dir(batch):
|
||||
if not attr_name.startswith("_") and not callable(
|
||||
getattr(batch, attr_name, None)
|
||||
):
|
||||
try:
|
||||
value = getattr(batch, attr_name, None)
|
||||
if value is not None and self._has_tensor(value):
|
||||
fixed_value = self._fix_tensor_device(
|
||||
value, target_device
|
||||
)
|
||||
setattr(batch, attr_name, fixed_value)
|
||||
except (AttributeError, TypeError):
|
||||
continue
|
||||
# DenoisingStage reads batch.scheduler. Native pipelines attach it in
|
||||
# TimestepPreparationStage; ComfyUI already owns the timestep schedule.
|
||||
get_or_create_request_scheduler(batch, self.scheduler)
|
||||
|
||||
original_latents_shape = None
|
||||
if batch.latents is not None:
|
||||
original_latents_shape = batch.latents.shape
|
||||
|
||||
# Call parent class's forward method
|
||||
result = super().forward(batch, server_args)
|
||||
|
||||
if original_latents_shape is not None:
|
||||
# Preserve the original shape before any potential packing/conversion
|
||||
# (e.g., 4D spatial -> 3D sequence) to ensure proper unpadding later.
|
||||
# Preserve the original shape before any packing/conversion
|
||||
# (e.g., 4D spatial -> 3D sequence) so unpadding stays correct.
|
||||
result.raw_latent_shape = original_latents_shape
|
||||
|
||||
return result
|
||||
|
||||
+2
@@ -3,6 +3,7 @@
|
||||
"""MiniMax H3-specific pipeline stages."""
|
||||
|
||||
from .stages.audio_encoding import MiniMaxH3AudioEncodingStage
|
||||
from .stages.comfyui_step import MiniMaxH3ComfyUIStepStage
|
||||
from .stages.decoding import MiniMaxH3DecodingStage
|
||||
from .stages.denoising import MiniMaxH3DenoisingStage
|
||||
from .stages.latent_preparation import MiniMaxH3LatentPreparationStage
|
||||
@@ -12,6 +13,7 @@ from .stages.visual_encoding import MiniMaxH3VisualEncodingStage
|
||||
|
||||
__all__ = [
|
||||
"MiniMaxH3AudioEncodingStage",
|
||||
"MiniMaxH3ComfyUIStepStage",
|
||||
"MiniMaxH3DecodingStage",
|
||||
"MiniMaxH3DenoisingStage",
|
||||
"MiniMaxH3LatentPreparationStage",
|
||||
|
||||
+799
@@ -0,0 +1,799 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""ComfyUI per-step DiT stage for MiniMax H3.
|
||||
|
||||
ComfyUI owns the sampler loop. This stage runs one ``MiniMaxH3DiTModel.forward``
|
||||
via ``MiniMaxH3DenoiseBranch.forward_kwargs`` and writes unpacked
|
||||
``[video, audio]`` velocities onto ``batch.noise_pred``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.minimax_h3 import (
|
||||
MINIMAX_H3_ADALN_MODALITY_NUM,
|
||||
MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.models.dits.minimax_h3 import (
|
||||
_FORWARD_SUPPORTED_KWARGS,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import (
|
||||
bind_comfyui_session,
|
||||
get_or_create_run_state,
|
||||
get_run_state,
|
||||
pop_run_state,
|
||||
release_run_state,
|
||||
session_id_from_req,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.schedule_batch import Req
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.base import PipelineStage
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.denoise_loop import (
|
||||
MINIMAX_H3_AUDIO_REF_COND_TIMESTEP,
|
||||
MINIMAX_H3_IMGVID_COND_TIMESTEP,
|
||||
MiniMaxH3DenoiseBranch,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.packed_sequence import (
|
||||
minimax_h3_packed_sequence,
|
||||
minimax_h3_packed_sequence_ref2va_blocks,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
|
||||
DEFAULT_SIGMA_SHIFT_VIDEO = 12.0
|
||||
DEFAULT_SIGMA_SHIFT_AUDIO = 3.0
|
||||
_PATCH_SIZE = (1, 2, 2)
|
||||
_SEG_TOKEN_TAG = {
|
||||
"text": 1,
|
||||
"video": 0,
|
||||
"cond": 0,
|
||||
"ref_img": 0,
|
||||
"audio": 2,
|
||||
"cond_audio": 2,
|
||||
"ref_audio": 2,
|
||||
}
|
||||
|
||||
|
||||
def time_shift_sigma(sigma, from_shift, to_shift):
|
||||
base = sigma / (from_shift + sigma * (1.0 - from_shift))
|
||||
return to_shift * base / (1.0 + (to_shift - 1.0) * base)
|
||||
|
||||
|
||||
def time_shift_slope(sigma, from_shift, to_shift):
|
||||
"""d(sigma_to)/d(sigma_from) at the same base-grid point."""
|
||||
base = sigma / (from_shift + sigma * (1.0 - from_shift))
|
||||
return (to_shift * (1.0 + (from_shift - 1.0) * base) ** 2) / (
|
||||
from_shift * (1.0 + (to_shift - 1.0) * base) ** 2
|
||||
)
|
||||
|
||||
|
||||
def patchify_video(latent, patch_size=_PATCH_SIZE):
|
||||
b, c, t_full, h_full, w_full = latent.shape
|
||||
pt, ph, pw = patch_size
|
||||
t, h, w = t_full // pt, h_full // ph, w_full // pw
|
||||
x = latent.reshape(b, c, t, pt, h, ph, w, pw)
|
||||
x = torch.einsum("nctrhpwq->nthwcrpq", x)
|
||||
return x.reshape(b * t * h * w, c * pt * ph * pw)
|
||||
|
||||
|
||||
def unpatchify_video(rows, t, h, w, c=24, patch_size=_PATCH_SIZE):
|
||||
pt, ph, pw = patch_size
|
||||
x = rows.reshape(-1, t, h, w, c, pt, ph, pw)
|
||||
x = torch.einsum("nthwcrpq->nctrhpwq", x)
|
||||
return x.reshape(-1, c, t * pt, h * ph, w * pw)
|
||||
|
||||
|
||||
def pack_audio(latent):
|
||||
_b, _c, ch, t = latent.shape
|
||||
return latent[0].permute(1, 2, 0).reshape(ch * t, latent.shape[1])
|
||||
|
||||
|
||||
def unpack_audio(rows, ch=2):
|
||||
t = rows.shape[0] // ch
|
||||
return rows.reshape(ch, t, rows.shape[-1]).permute(2, 0, 1).unsqueeze(0)
|
||||
|
||||
|
||||
def pad_to_patch_size(video, patch_size=_PATCH_SIZE):
|
||||
pt, ph, pw = patch_size
|
||||
t, h, w = video.shape[2], video.shape[3], video.shape[4]
|
||||
pad_t = (pt - t % pt) % pt
|
||||
pad_h = (ph - h % ph) % ph
|
||||
pad_w = (pw - w % pw) % pw
|
||||
if pad_t == pad_h == pad_w == 0:
|
||||
return video
|
||||
return torch.nn.functional.pad(video, (0, pad_w, 0, pad_h, 0, pad_t))
|
||||
|
||||
|
||||
def _as_text_embeddings(context: torch.Tensor) -> torch.Tensor:
|
||||
if context.ndim == 3:
|
||||
return context[0]
|
||||
return context
|
||||
|
||||
|
||||
def _payload_has_keyframes(payload: dict[str, Any] | None) -> bool:
|
||||
return bool((payload or {}).get("keyframes"))
|
||||
|
||||
|
||||
def _payload_refs(payload: dict[str, Any] | None) -> list[dict[str, Any]]:
|
||||
refs = (payload or {}).get("refs") or []
|
||||
if not isinstance(refs, (list, tuple)):
|
||||
raise TypeError(f"minimax_payload.refs must be a sequence, got {type(refs)!r}")
|
||||
return list(refs)
|
||||
|
||||
|
||||
def _apply_cond_noise(rows: torch.Tensor, aug: float, seed: int) -> torch.Tensor:
|
||||
if aug >= 1.0:
|
||||
return rows
|
||||
gen = torch.Generator("cpu").manual_seed(int(seed))
|
||||
noise = torch.randn(rows.shape, generator=gen, dtype=torch.float32)
|
||||
return aug * rows + (1.0 - aug) * noise.to(rows.device)
|
||||
|
||||
|
||||
def _ref_blocks_for_packed(refs: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
blocks: list[dict[str, Any]] = []
|
||||
for ref in refs:
|
||||
block = {
|
||||
key: value
|
||||
for key, value in ref.items()
|
||||
if key not in {"latent", "audio_latent"}
|
||||
}
|
||||
if "kind" not in block and "type" in block:
|
||||
block["kind"] = block["type"]
|
||||
blocks.append(block)
|
||||
return blocks
|
||||
|
||||
|
||||
def serialize_comfyui_layout(layout: Any) -> dict[str, Any] | None:
|
||||
"""Turn a ComfyUI ``PackedLayout`` into pickle/IPC-safe tensors."""
|
||||
required = (
|
||||
"seq_len",
|
||||
"signature",
|
||||
"segments",
|
||||
"position_ids",
|
||||
"img_pos",
|
||||
"img_update",
|
||||
"audio_pos",
|
||||
"audio_update",
|
||||
)
|
||||
if layout is None or any(not hasattr(layout, name) for name in required):
|
||||
return None
|
||||
return {
|
||||
"seq_len": int(layout.seq_len),
|
||||
"signature": tuple(int(value) for value in layout.signature),
|
||||
"segments": [
|
||||
(int(start), int(stop), str(kind)) for start, stop, kind in layout.segments
|
||||
],
|
||||
"position_ids": layout.position_ids.detach().cpu().to(torch.float64),
|
||||
"img_pos": layout.img_pos.detach().cpu().to(torch.long),
|
||||
"img_update": layout.img_update.detach().cpu().to(torch.bool),
|
||||
"audio_pos": layout.audio_pos.detach().cpu().to(torch.long),
|
||||
"audio_update": layout.audio_update.detach().cpu().to(torch.bool),
|
||||
}
|
||||
|
||||
|
||||
def _layout_signature_matches(
|
||||
layout: dict[str, Any],
|
||||
video_x: torch.Tensor,
|
||||
audio_x: torch.Tensor,
|
||||
text: torch.Tensor,
|
||||
) -> bool:
|
||||
"""ComfyUI ``PackedLayout.signature`` is ``(text, T, H, W, audio_t)``.
|
||||
|
||||
A cached layout from an earlier sampler pass (e.g. pre-upscale 14×8
|
||||
tokens) must not be applied to a different spatial grid.
|
||||
"""
|
||||
sig = layout.get("signature")
|
||||
if sig is None:
|
||||
return False
|
||||
values = tuple(int(v) for v in sig[:5])
|
||||
if len(values) < 5:
|
||||
return False
|
||||
text_len, latent_t, latent_h, latent_w, audio_t = values
|
||||
return (
|
||||
text_len == int(text.shape[0])
|
||||
and latent_t == int(video_x.shape[2])
|
||||
and latent_h == int(video_x.shape[3])
|
||||
and latent_w == int(video_x.shape[4])
|
||||
and audio_t == int(audio_x.shape[-1])
|
||||
)
|
||||
|
||||
|
||||
def comfyui_layout_to_packed(layout: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Rebuild the SGLD packed dict from a serialized ComfyUI layout.
|
||||
|
||||
Only pads ``seq_len`` to 64. Row order, RoPE ids, and update masks stay
|
||||
exactly as ComfyUI computed them.
|
||||
"""
|
||||
used = int(layout["seq_len"])
|
||||
seq_len = (
|
||||
(used + MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT - 1)
|
||||
// MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT
|
||||
* MINIMAX_H3_PACKED_SEQUENCE_ALIGNMENT
|
||||
)
|
||||
position_ids = layout["position_ids"].to(dtype=torch.float64)
|
||||
grid = torch.zeros(seq_len, 3, dtype=torch.float64)
|
||||
grid[:used] = position_ids[:used]
|
||||
token_tags = torch.full((seq_len,), -1, dtype=torch.long)
|
||||
text_pos = torch.empty(0, dtype=torch.long)
|
||||
for start, stop, kind in layout["segments"]:
|
||||
token_tags[start:stop] = _SEG_TOKEN_TAG.get(kind, -1)
|
||||
if kind == "text":
|
||||
text_pos = torch.arange(start, stop, dtype=torch.long)
|
||||
return {
|
||||
"seq_len": seq_len,
|
||||
"img_pos": layout["img_pos"].view(-1).to(torch.long),
|
||||
"audio_pos": layout["audio_pos"].view(-1).to(torch.long),
|
||||
"text_pos": text_pos,
|
||||
"update_mask": layout["img_update"].view(-1).to(torch.bool),
|
||||
"audio_update_mask": layout["audio_update"].view(-1).to(torch.bool),
|
||||
"img_position_ids": grid,
|
||||
"token_tags": token_tags,
|
||||
"cu_seqlens": torch.tensor([0, used, seq_len], dtype=torch.int32),
|
||||
}
|
||||
|
||||
|
||||
def _latent_items(payload: dict[str, Any], key: str) -> list[Any]:
|
||||
items = payload.get(key) or []
|
||||
if not isinstance(items, (list, tuple)):
|
||||
return [items]
|
||||
return [item for item in items if item is not None]
|
||||
|
||||
|
||||
def _assemble_media_rows(
|
||||
video: torch.Tensor,
|
||||
audio: torch.Tensor,
|
||||
payload: dict[str, Any],
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
video_target = patchify_video(video, _PATCH_SIZE)
|
||||
audio_target = pack_audio(audio)
|
||||
vis_aug = float(
|
||||
payload.get("visual_cond_noise_aug", MINIMAX_H3_IMGVID_COND_TIMESTEP)
|
||||
)
|
||||
aud_aug = float(
|
||||
payload.get("audio_cond_noise_aug", MINIMAX_H3_AUDIO_REF_COND_TIMESTEP)
|
||||
)
|
||||
seed = int(payload.get("seed", 0))
|
||||
|
||||
cond_video_src = list(_latent_items(payload, "cond_video_latents"))
|
||||
cond_audio_src = list(_latent_items(payload, "cond_audio_latents"))
|
||||
if not cond_video_src and not cond_audio_src:
|
||||
for block in list(payload.get("keyframes") or []) + _payload_refs(payload):
|
||||
if block.get("latent") is not None:
|
||||
cond_video_src.append(block["latent"])
|
||||
if block.get("audio_latent") is not None:
|
||||
cond_audio_src.append(block["audio_latent"])
|
||||
|
||||
cond_video: list[torch.Tensor] = []
|
||||
for latent in cond_video_src:
|
||||
rows = patchify_video(
|
||||
pad_to_patch_size(
|
||||
latent.to(device=video.device, dtype=torch.float32),
|
||||
_PATCH_SIZE,
|
||||
),
|
||||
_PATCH_SIZE,
|
||||
)
|
||||
cond_video.append(_apply_cond_noise(rows, vis_aug, seed))
|
||||
cond_audio: list[torch.Tensor] = []
|
||||
for latent in cond_audio_src:
|
||||
rows = pack_audio(latent.to(device=audio.device, dtype=torch.float32))
|
||||
cond_audio.append(_apply_cond_noise(rows, aud_aug, seed + 1))
|
||||
|
||||
video_rows = (
|
||||
torch.cat(cond_video + [video_target], dim=0) if cond_video else video_target
|
||||
)
|
||||
audio_rows = (
|
||||
torch.cat(cond_audio + [audio_target], dim=0) if cond_audio else audio_target
|
||||
)
|
||||
return video_rows, audio_rows
|
||||
|
||||
|
||||
def _token_tags_for_payload(
|
||||
packed: dict[str, torch.Tensor],
|
||||
payload: dict[str, Any],
|
||||
) -> torch.Tensor:
|
||||
tags = packed["token_tags"]
|
||||
text_tags = payload.get("text_token_tags")
|
||||
if text_tags is None:
|
||||
return tags
|
||||
text_pos = packed["text_pos"].view(-1)
|
||||
text_tags = text_tags.reshape(-1).to(dtype=torch.long)
|
||||
if int(text_tags.numel()) != int(text_pos.numel()):
|
||||
raise ValueError(
|
||||
"text_token_tags length "
|
||||
f"{int(text_tags.numel())} != packed text_len {int(text_pos.numel())}"
|
||||
)
|
||||
tags = tags.clone()
|
||||
tags[text_pos] = text_tags
|
||||
return tags
|
||||
|
||||
|
||||
def _sigma_list(sample_sigmas: Any) -> list[float] | None:
|
||||
n = _sample_sigmas_len(sample_sigmas)
|
||||
if n is None:
|
||||
return None
|
||||
if torch.is_tensor(sample_sigmas):
|
||||
return [float(v) for v in sample_sigmas.reshape(-1).tolist()]
|
||||
return [float(v) for v in sample_sigmas]
|
||||
|
||||
|
||||
def _as_spatial_mask(mask: Any) -> torch.Tensor | None:
|
||||
if mask is None:
|
||||
return None
|
||||
values = mask
|
||||
if not torch.is_tensor(values):
|
||||
values = torch.as_tensor(values)
|
||||
if int(values.numel()) == 0:
|
||||
return None
|
||||
if values.ndim == 5:
|
||||
values = values[0, 0]
|
||||
elif values.ndim == 4:
|
||||
values = values[0]
|
||||
elif values.ndim == 2:
|
||||
values = values.unsqueeze(0)
|
||||
return values.to(dtype=torch.float32)
|
||||
|
||||
|
||||
def mask_row_values(mask: Any, latent_t: int, lat_h: int, lat_w: int):
|
||||
"""Match ComfyUI ``mask_row_values``: [T,H,W] → one float per 2×2 patch row."""
|
||||
values = _as_spatial_mask(mask)
|
||||
if values is None:
|
||||
return None
|
||||
if values.ndim != 3:
|
||||
raise ValueError(
|
||||
f"denoise_mask must be [T,H,W] after squeeze, got {tuple(values.shape)}"
|
||||
)
|
||||
if values.shape[0] == 1 and latent_t > 1:
|
||||
values = values.expand(latent_t, *values.shape[1:])
|
||||
elif values.shape[0] != latent_t:
|
||||
raise ValueError(
|
||||
f"denoise_mask T={int(values.shape[0])} != padded latent T={latent_t}"
|
||||
)
|
||||
pad_w = lat_w - int(values.shape[-1])
|
||||
pad_h = lat_h - int(values.shape[-2])
|
||||
if pad_w > 0 or pad_h > 0:
|
||||
values = torch.nn.functional.pad(
|
||||
values, (0, max(pad_w, 0), 0, max(pad_h, 0)), mode="replicate"
|
||||
)
|
||||
values = values[:latent_t, :lat_h, :lat_w]
|
||||
values = values.reshape(latent_t, lat_h // 2, 2, lat_w // 2, 2).amax(dim=(2, 4))
|
||||
rows = values.reshape(-1)
|
||||
if bool((rows >= 1.0 - 1e-3).all()):
|
||||
return None
|
||||
return rows
|
||||
|
||||
|
||||
def _audio_mask_values(mask: Any):
|
||||
if mask is None:
|
||||
return None
|
||||
values = mask.reshape(-1).to(dtype=torch.float32)
|
||||
if int(values.numel()) == 0:
|
||||
return None
|
||||
if bool((values >= 1.0 - 1e-3).all()):
|
||||
return None
|
||||
return values
|
||||
|
||||
|
||||
def _overlay_per_row_timesteps(
|
||||
fk: dict[str, Any],
|
||||
branch: MiniMaxH3DenoiseBranch,
|
||||
video_rows_t: torch.Tensor | None,
|
||||
audio_rows_t: torch.Tensor | None,
|
||||
) -> dict[str, Any]:
|
||||
"""Apply ComfyUI per-row denoise_mask only on target video/audio rows."""
|
||||
if video_rows_t is None and audio_rows_t is None:
|
||||
return fk
|
||||
unique = fk["unique_timesteps"].to(torch.float32)
|
||||
inverse = fk["inverse_indices"].clone()
|
||||
device = inverse.device
|
||||
extras = [unique.reshape(-1).detach().cpu()]
|
||||
if video_rows_t is not None:
|
||||
extras.append(video_rows_t.reshape(-1).to(torch.float32).cpu())
|
||||
if audio_rows_t is not None:
|
||||
extras.append(audio_rows_t.reshape(-1).to(torch.float32).cpu())
|
||||
new_unique = torch.unique(torch.cat(extras), sorted=True)
|
||||
inverse = torch.searchsorted(new_unique, unique.cpu()).to(device)[inverse]
|
||||
if video_rows_t is not None:
|
||||
mapped = torch.searchsorted(
|
||||
new_unique, video_rows_t.reshape(-1).to(torch.float32).cpu()
|
||||
)
|
||||
tgt = branch.img_target_seq_idx
|
||||
if int(mapped.numel()) != int(tgt.numel()):
|
||||
raise ValueError(
|
||||
"denoise_mask rows "
|
||||
f"{int(mapped.numel())} != video target rows {int(tgt.numel())}"
|
||||
)
|
||||
inverse[tgt] = mapped.to(device)
|
||||
if audio_rows_t is not None:
|
||||
mapped = torch.searchsorted(
|
||||
new_unique, audio_rows_t.reshape(-1).to(torch.float32).cpu()
|
||||
)
|
||||
tgt = branch.audio_target_seq_idx
|
||||
if int(mapped.numel()) != int(tgt.numel()):
|
||||
raise ValueError(
|
||||
"audio_denoise_mask rows "
|
||||
f"{int(mapped.numel())} != audio target rows {int(tgt.numel())}"
|
||||
)
|
||||
inverse[tgt] = mapped.to(device)
|
||||
fk["unique_timesteps"] = new_unique.to(device=device, dtype=unique.dtype)
|
||||
fk["inverse_indices"] = inverse
|
||||
fk["block_combined_indices"] = (
|
||||
branch.block_token_tags
|
||||
+ inverse[branch.local_row_slice] * MINIMAX_H3_ADALN_MODALITY_NUM
|
||||
)
|
||||
return fk
|
||||
|
||||
|
||||
def comfyui_payload_to_branch_inputs(
|
||||
video_x: torch.Tensor,
|
||||
audio_x: torch.Tensor,
|
||||
context: torch.Tensor,
|
||||
minimax_payload: dict[str, Any] | None,
|
||||
sample_sigmas: Any,
|
||||
timestep: torch.Tensor,
|
||||
transformer_options: dict[str, Any] | None = None,
|
||||
layout: dict[str, Any] | None = None,
|
||||
denoise_mask: Any = None,
|
||||
audio_denoise_mask: Any = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Translate one ComfyUI H3 step into Branch + ``forward_kwargs`` inputs.
|
||||
|
||||
``context`` is raw Qwen states (last dim 5120). Token refinement is
|
||||
precomputed once on the worker after the Branch is created.
|
||||
"""
|
||||
payload = minimax_payload or {}
|
||||
orig_shape = tuple(video_x.shape)
|
||||
video = pad_to_patch_size(video_x.to(torch.float32), _PATCH_SIZE)
|
||||
audio = audio_x.to(torch.float32)
|
||||
text = _as_text_embeddings(context)
|
||||
if layout is not None and not _layout_signature_matches(
|
||||
layout, video_x, audio, text
|
||||
):
|
||||
_logger.warning(
|
||||
"Ignoring stale ComfyUI PackedLayout signature %s for video %s "
|
||||
"audio %s text_len %s",
|
||||
layout.get("signature"),
|
||||
tuple(video_x.shape),
|
||||
tuple(audio.shape),
|
||||
int(text.shape[0]),
|
||||
)
|
||||
layout = None
|
||||
if layout is None and _payload_has_keyframes(payload):
|
||||
raise NotImplementedError(
|
||||
"MiniMaxH3 ComfyUI step needs a serialized PackedLayout for fl2va keyframes"
|
||||
)
|
||||
text_len = int(text.shape[0])
|
||||
latent_t, latent_h, latent_w = (
|
||||
int(video.shape[2]),
|
||||
int(video.shape[3]),
|
||||
int(video.shape[4]),
|
||||
)
|
||||
audio_t = int(audio.shape[-1])
|
||||
refs = _payload_refs(payload)
|
||||
if layout is not None:
|
||||
packed = comfyui_layout_to_packed(layout)
|
||||
elif refs:
|
||||
packed = minimax_h3_packed_sequence_ref2va_blocks(
|
||||
text_len=text_len,
|
||||
latent_t=latent_t,
|
||||
latent_h=latent_h,
|
||||
latent_w=latent_w,
|
||||
audio_t=audio_t,
|
||||
ref_blocks=_ref_blocks_for_packed(refs),
|
||||
)
|
||||
else:
|
||||
packed = minimax_h3_packed_sequence(
|
||||
text_len=text_len,
|
||||
latent_t=latent_t,
|
||||
latent_h=latent_h,
|
||||
latent_w=latent_w,
|
||||
audio_t=audio_t,
|
||||
include_keyframe_cond=False,
|
||||
)
|
||||
used = int(packed["cu_seqlens"][1])
|
||||
video_rows, audio_rows = _assemble_media_rows(video, audio, payload)
|
||||
|
||||
opts = transformer_options or {}
|
||||
shift_v = float(opts.get("minimax_h3_sigma_shift_video", DEFAULT_SIGMA_SHIFT_VIDEO))
|
||||
shift_a = float(opts.get("minimax_h3_sigma_shift_audio", DEFAULT_SIGMA_SHIFT_AUDIO))
|
||||
sigma_v = (timestep.reshape(-1)[0] / 1000.0).to(torch.float32).clamp(min=1e-6)
|
||||
sigma_a = time_shift_sigma(sigma_v, shift_v, shift_a)
|
||||
t_video = float(1.0 - sigma_v)
|
||||
t_audio = float(1.0 - sigma_a)
|
||||
t_pin_v = max(t_video, MINIMAX_H3_IMGVID_COND_TIMESTEP)
|
||||
t_pin_a = max(t_audio, MINIMAX_H3_AUDIO_REF_COND_TIMESTEP)
|
||||
# ComfyUI only remaps the video/audio *target* segments. Text / pad keep t_video.
|
||||
video_rows_t = mask_row_values(denoise_mask, latent_t, latent_h, latent_w)
|
||||
if video_rows_t is not None:
|
||||
video_rows_t = (1.0 - video_rows_t * float(sigma_v)).clamp(max=t_pin_v)
|
||||
audio_rows_t = _audio_mask_values(audio_denoise_mask)
|
||||
if audio_rows_t is not None:
|
||||
audio_rows_t = (1.0 - audio_rows_t * float(sigma_a)).clamp(max=t_pin_a)
|
||||
|
||||
return {
|
||||
"packed": packed,
|
||||
"text_embeddings": text,
|
||||
"token_tags": _token_tags_for_payload(packed, payload),
|
||||
"video_rows": video_rows,
|
||||
"audio_rows": audio_rows,
|
||||
"t_video": t_video,
|
||||
"t_audio": t_audio,
|
||||
"video_rows_t": video_rows_t,
|
||||
"audio_rows_t": audio_rows_t,
|
||||
"used": used,
|
||||
"orig_video_shape": orig_shape,
|
||||
"padded_video_shape": tuple(video.shape),
|
||||
"sample_sigmas": sample_sigmas,
|
||||
"sigma_v": float(sigma_v),
|
||||
"shift_v": shift_v,
|
||||
"shift_a": shift_a,
|
||||
"imgvid_cond_noise_aug": float(
|
||||
payload.get("visual_cond_noise_aug", MINIMAX_H3_IMGVID_COND_TIMESTEP)
|
||||
),
|
||||
"audio_ref_cond_noise_aug": float(
|
||||
payload.get("audio_cond_noise_aug", MINIMAX_H3_AUDIO_REF_COND_TIMESTEP)
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def build_step_forward_kwargs(
|
||||
inputs: dict[str, Any],
|
||||
*,
|
||||
branch: MiniMaxH3DenoiseBranch | None = None,
|
||||
device: torch.device | None = None,
|
||||
) -> tuple[dict[str, Any], MiniMaxH3DenoiseBranch]:
|
||||
"""Assemble DiT ``forward`` kwargs from ``comfyui_payload_to_branch_inputs``."""
|
||||
if device is None:
|
||||
device = inputs["video_rows"].device
|
||||
if branch is None:
|
||||
branch = MiniMaxH3DenoiseBranch(
|
||||
packed=inputs["packed"],
|
||||
text_embeddings=inputs["text_embeddings"],
|
||||
token_tags=inputs["token_tags"],
|
||||
device=device,
|
||||
)
|
||||
plan = branch.prepare_timestep_plan(
|
||||
video_timesteps=[inputs["t_video"]],
|
||||
audio_timesteps=[inputs["t_audio"]],
|
||||
imgvid_cond_noise_aug=inputs["imgvid_cond_noise_aug"],
|
||||
audio_ref_cond_noise_aug=inputs["audio_ref_cond_noise_aug"],
|
||||
)
|
||||
fk = branch.forward_kwargs(
|
||||
video_rows=inputs["video_rows"].to(device=device, dtype=torch.float32),
|
||||
audio_rows=inputs["audio_rows"].to(device=device, dtype=torch.float32),
|
||||
step_timesteps=plan[0],
|
||||
)
|
||||
fk = _overlay_per_row_timesteps(
|
||||
fk, branch, inputs.get("video_rows_t"), inputs.get("audio_rows_t")
|
||||
)
|
||||
extra = set(fk) - _FORWARD_SUPPORTED_KWARGS
|
||||
if extra:
|
||||
raise TypeError(
|
||||
f"MiniMaxH3 ComfyUI step produced unsupported forward kwargs: {sorted(extra)}"
|
||||
)
|
||||
return fk, branch
|
||||
|
||||
|
||||
def pack_comfy_output(
|
||||
v_video: torch.Tensor,
|
||||
v_audio: torch.Tensor,
|
||||
state: MiniMaxH3ComfyUIRunState,
|
||||
) -> list[torch.Tensor]:
|
||||
_b, _c, orig_t, orig_h, orig_w = state.orig_video_shape
|
||||
_pb, _pc, padded_t, padded_h, padded_w = state.padded_video_shape
|
||||
video = unpatchify_video(
|
||||
v_video,
|
||||
padded_t,
|
||||
padded_h // 2,
|
||||
padded_w // 2,
|
||||
c=int(state.orig_video_shape[1]),
|
||||
patch_size=_PATCH_SIZE,
|
||||
)
|
||||
video = video[:, :, :orig_t, :orig_h, :orig_w]
|
||||
# DiT returns target video rows, but audio_pos includes reference rows.
|
||||
audio = unpack_audio(v_audio[state.branch.audio_target_slice])
|
||||
return [video, audio]
|
||||
|
||||
|
||||
def _sample_sigmas_len(sample_sigmas: Any) -> int | None:
|
||||
if sample_sigmas is None:
|
||||
return None
|
||||
if torch.is_tensor(sample_sigmas):
|
||||
return int(sample_sigmas.reshape(-1).numel())
|
||||
if isinstance(sample_sigmas, (list, tuple)):
|
||||
return len(sample_sigmas)
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class MiniMaxH3ComfyUIRunState:
|
||||
branch: MiniMaxH3DenoiseBranch
|
||||
used: int
|
||||
orig_video_shape: tuple[int, ...]
|
||||
padded_video_shape: tuple[int, ...]
|
||||
sample_sigmas: Any = None
|
||||
cache_mounted: bool = False
|
||||
refined: bool = False
|
||||
steps_done: int = 0
|
||||
signature: tuple[Any, ...] | None = None
|
||||
|
||||
|
||||
class MiniMaxH3ComfyUIStepStage(PipelineStage):
|
||||
def __init__(self, transformer) -> None:
|
||||
super().__init__()
|
||||
self.transformer = transformer
|
||||
self._cache_stage = None
|
||||
|
||||
def verify_input(self, batch: Req, server_args: ServerArgs):
|
||||
bind_comfyui_session(batch)
|
||||
return super().verify_input(batch, server_args)
|
||||
|
||||
def _cache_dit_stage(self):
|
||||
if self._cache_stage is None:
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.denoising import (
|
||||
MiniMaxH3DenoisingStage,
|
||||
)
|
||||
|
||||
self._cache_stage = MiniMaxH3DenoisingStage(transformer=self.transformer)
|
||||
return self._cache_stage
|
||||
|
||||
def _maybe_mount_cache_dit(
|
||||
self, batch: Req, state: MiniMaxH3ComfyUIRunState
|
||||
) -> None:
|
||||
if state.cache_mounted:
|
||||
return
|
||||
n_sigmas = _sample_sigmas_len(state.sample_sigmas)
|
||||
if n_sigmas is None or n_sigmas < 2:
|
||||
return
|
||||
self._cache_dit_stage()._maybe_enable_cache_dit(n_sigmas - 1, batch)
|
||||
state.cache_mounted = True
|
||||
|
||||
def _release_run_state(self, state: MiniMaxH3ComfyUIRunState | None) -> None:
|
||||
if state is None:
|
||||
return
|
||||
if state.cache_mounted and self._cache_stage is not None:
|
||||
self._cache_stage._unmount_cache_dit()
|
||||
state.cache_mounted = False
|
||||
state.branch = None
|
||||
|
||||
def _read_step_tensors(
|
||||
self, batch: Req
|
||||
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
|
||||
extra = getattr(batch, "extra", None) or {}
|
||||
video = batch.latents
|
||||
audio = batch.audio_latents
|
||||
if audio is None:
|
||||
audio = extra.get("audio_latents")
|
||||
context = extra.get("h3_context")
|
||||
if context is None:
|
||||
embeds = batch.prompt_embeds
|
||||
if isinstance(embeds, list) and embeds:
|
||||
context = embeds[0]
|
||||
else:
|
||||
context = embeds
|
||||
if video is None or audio is None or context is None:
|
||||
raise ValueError(
|
||||
"MiniMaxH3 ComfyUI step requires latents, audio_latents, and prompt_embeds"
|
||||
)
|
||||
return video, audio, context
|
||||
|
||||
def _bind_run_state(
|
||||
self,
|
||||
batch: Req,
|
||||
inputs: dict[str, Any],
|
||||
sample_sigmas: Any,
|
||||
device: torch.device,
|
||||
) -> MiniMaxH3ComfyUIRunState:
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.stages.denoising import (
|
||||
_precompute_refined_prompt_embeds,
|
||||
)
|
||||
|
||||
signature = (
|
||||
inputs["orig_video_shape"],
|
||||
inputs["used"],
|
||||
int(inputs["text_embeddings"].shape[0]),
|
||||
)
|
||||
existing = get_run_state(batch)
|
||||
sigmas = _sigma_list(sample_sigmas)
|
||||
is_first_sigma = False
|
||||
if sigmas:
|
||||
is_first_sigma = abs(float(inputs["sigma_v"]) - sigmas[0]) < 1e-4
|
||||
if existing is not None and (
|
||||
existing.signature != signature
|
||||
or (is_first_sigma and existing.steps_done > 0)
|
||||
):
|
||||
self._release_run_state(existing)
|
||||
pop_run_state(batch)
|
||||
existing = None
|
||||
|
||||
def _factory() -> MiniMaxH3ComfyUIRunState:
|
||||
branch = MiniMaxH3DenoiseBranch(
|
||||
packed=inputs["packed"],
|
||||
text_embeddings=inputs["text_embeddings"],
|
||||
token_tags=inputs["token_tags"],
|
||||
device=device,
|
||||
)
|
||||
return MiniMaxH3ComfyUIRunState(
|
||||
branch=branch,
|
||||
used=inputs["used"],
|
||||
orig_video_shape=inputs["orig_video_shape"],
|
||||
padded_video_shape=inputs["padded_video_shape"],
|
||||
sample_sigmas=sample_sigmas,
|
||||
signature=signature,
|
||||
)
|
||||
|
||||
state = (
|
||||
existing
|
||||
if existing is not None
|
||||
else get_or_create_run_state(batch, _factory)
|
||||
)
|
||||
if not state.refined:
|
||||
_precompute_refined_prompt_embeds(
|
||||
self.transformer, state.branch, device=device
|
||||
)
|
||||
state.refined = True
|
||||
return state
|
||||
|
||||
def forward(self, batch: Req, server_args: ServerArgs) -> Req:
|
||||
bind_comfyui_session(batch)
|
||||
extra = getattr(batch, "extra", None) or {}
|
||||
video_x, audio_x, context = self._read_step_tensors(batch)
|
||||
payload = extra.get("h3_payload") or {}
|
||||
transformer_options = extra.get("h3_transformer_options") or {}
|
||||
sample_sigmas = extra.get("h3_sample_sigmas")
|
||||
if sample_sigmas is None:
|
||||
sample_sigmas = batch.sigmas
|
||||
timestep = batch.timesteps
|
||||
if timestep is None:
|
||||
timestep = batch.timestep
|
||||
if timestep is None:
|
||||
raise ValueError("MiniMaxH3 ComfyUI step requires timesteps")
|
||||
|
||||
inputs = comfyui_payload_to_branch_inputs(
|
||||
video_x,
|
||||
audio_x,
|
||||
context,
|
||||
payload,
|
||||
sample_sigmas,
|
||||
timestep,
|
||||
transformer_options=transformer_options,
|
||||
layout=extra.get("h3_layout"),
|
||||
denoise_mask=extra.get("h3_denoise_mask"),
|
||||
audio_denoise_mask=extra.get("h3_audio_denoise_mask"),
|
||||
)
|
||||
state = self._bind_run_state(batch, inputs, sample_sigmas, video_x.device)
|
||||
self._maybe_mount_cache_dit(batch, state)
|
||||
fk, _branch = build_step_forward_kwargs(
|
||||
inputs, branch=state.branch, device=video_x.device
|
||||
)
|
||||
v_video, v_audio = self.transformer(**fk)
|
||||
batch.noise_pred = pack_comfy_output(v_video, v_audio, state)
|
||||
state.steps_done += 1
|
||||
n_sigmas = _sample_sigmas_len(sample_sigmas)
|
||||
if n_sigmas is not None and n_sigmas >= 2 and state.steps_done >= n_sigmas - 1:
|
||||
self._release_run_state(state)
|
||||
release_run_state(session_id_from_req(batch))
|
||||
return batch
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DEFAULT_SIGMA_SHIFT_AUDIO",
|
||||
"DEFAULT_SIGMA_SHIFT_VIDEO",
|
||||
"MiniMaxH3ComfyUIRunState",
|
||||
"MiniMaxH3ComfyUIStepStage",
|
||||
"build_step_forward_kwargs",
|
||||
"comfyui_layout_to_packed",
|
||||
"comfyui_payload_to_branch_inputs",
|
||||
"mask_row_values",
|
||||
"pack_audio",
|
||||
"pack_comfy_output",
|
||||
"patchify_video",
|
||||
"serialize_comfyui_layout",
|
||||
"time_shift_sigma",
|
||||
"time_shift_slope",
|
||||
"unpack_audio",
|
||||
"unpatchify_video",
|
||||
]
|
||||
@@ -7,6 +7,11 @@ from typing import Any, Optional
|
||||
import zmq
|
||||
import zmq.asyncio
|
||||
|
||||
from sglang.multimodal_gen.runtime.distributed.ipc_cuda import (
|
||||
materialize_cuda_refs,
|
||||
release_retained_producer_tensors,
|
||||
spill_cuda_tensors,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.entrypoints.control_requests import (
|
||||
ListLorasReq,
|
||||
MergeLoraWeightsReq,
|
||||
@@ -22,7 +27,10 @@ from sglang.multimodal_gen.runtime.entrypoints.post_training.io_struct import (
|
||||
UpdateWeightFromTensorCheckerReqInput,
|
||||
UpdateWeightFromTensorReqInput,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.ipc_array import materialize_file_refs
|
||||
from sglang.multimodal_gen.runtime.ipc_array import (
|
||||
is_local_endpoint,
|
||||
materialize_file_refs,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core import Req
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.schedule_batch import OutputBatch
|
||||
from sglang.multimodal_gen.runtime.server_args import (
|
||||
@@ -174,9 +182,10 @@ class SchedulerClient:
|
||||
effective_timeout = _resolve_timeout_ms(self.server_args, timeout_ms)
|
||||
_configure_recv_timeout(socket, effective_timeout)
|
||||
socket.connect(endpoint)
|
||||
socket.send_pyobj(batch)
|
||||
socket.send_pyobj(_prepare_local_payload(endpoint, batch))
|
||||
output_batch = socket.recv_pyobj()
|
||||
_materialize_output_batch_file_refs(output_batch)
|
||||
_materialize_local_cuda_refs(endpoint, output_batch)
|
||||
return output_batch
|
||||
except zmq.error.Again:
|
||||
logger.error("Timeout waiting for response from %s.", endpoint)
|
||||
@@ -286,10 +295,11 @@ class AsyncSchedulerClient:
|
||||
effective_timeout = _resolve_timeout_ms(self.server_args, timeout_ms)
|
||||
_configure_recv_timeout(socket, effective_timeout)
|
||||
socket.connect(endpoint)
|
||||
await socket.send(pickle.dumps(batch))
|
||||
await socket.send(pickle.dumps(_prepare_local_payload(endpoint, batch)))
|
||||
payload = await socket.recv()
|
||||
output_batch = pickle.loads(payload)
|
||||
_materialize_output_batch_file_refs(output_batch)
|
||||
_materialize_local_cuda_refs(endpoint, output_batch)
|
||||
return output_batch
|
||||
except zmq.error.Again:
|
||||
logger.error("Timeout waiting for response from %s.", endpoint)
|
||||
@@ -331,6 +341,19 @@ async_scheduler_client = AsyncSchedulerClient()
|
||||
sync_scheduler_client = SchedulerClient()
|
||||
|
||||
|
||||
def _prepare_local_payload(endpoint: str, batch: Any) -> Any:
|
||||
if is_local_endpoint(endpoint):
|
||||
return spill_cuda_tensors(batch)
|
||||
return batch
|
||||
|
||||
|
||||
def _materialize_local_cuda_refs(endpoint: str, output_batch: Any) -> None:
|
||||
if is_local_endpoint(endpoint):
|
||||
materialize_cuda_refs(output_batch)
|
||||
# Request-side IPC retains are no longer needed after the round-trip.
|
||||
release_retained_producer_tensors()
|
||||
|
||||
|
||||
def _materialize_output_batch_file_refs(output_batch: Any) -> None:
|
||||
if not isinstance(output_batch, OutputBatch):
|
||||
return
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Pack / unpack contract for ComfyUI model adapters."""
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.adapter import (
|
||||
get_adapter_class,
|
||||
registered_model_types,
|
||||
)
|
||||
from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.flux import FluxAdapter
|
||||
from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.executors.zimage import (
|
||||
ZImageAdapter,
|
||||
)
|
||||
|
||||
|
||||
def test_registered_comfyui_model_types() -> None:
|
||||
types = registered_model_types()
|
||||
assert "flux" in types
|
||||
assert "lumina2" in types
|
||||
assert get_adapter_class("lumina2") is ZImageAdapter
|
||||
assert get_adapter_class("flux") is FluxAdapter
|
||||
assert get_adapter_class("lumina2").pipeline_class_name == "ZImagePipeline"
|
||||
|
||||
|
||||
def test_zimage_pack_sets_seq_lens_and_time_dim() -> None:
|
||||
adapter = ZImageAdapter()
|
||||
x = torch.ones(1, 16, 90, 160)
|
||||
timestep = torch.tensor([1.0])
|
||||
context = torch.ones(1, 19, 2560)
|
||||
packed = adapter.pack(x, timestep, context)
|
||||
assert packed.latents.shape == (1, 16, 1, 90, 160)
|
||||
assert packed.prompt_embeds[0].shape == (19, 2560)
|
||||
assert packed.prompt_seq_lens == [[19]]
|
||||
assert packed.height == 720
|
||||
assert packed.width == 1280
|
||||
assert torch.equal(packed.timesteps, timestep * 1000.0)
|
||||
|
||||
pred = torch.ones(1, 16, 1, 90, 160)
|
||||
out = adapter.unpack(pred, packed, x)
|
||||
assert out.shape == x.shape
|
||||
|
||||
|
||||
def test_flux_pack_and_unpack_roundtrip() -> None:
|
||||
adapter = FluxAdapter()
|
||||
x = torch.arange(1 * 16 * 8 * 8, dtype=torch.float32).reshape(1, 16, 8, 8)
|
||||
timestep = torch.tensor([0.5])
|
||||
context = torch.ones(1, 8, 4096)
|
||||
y = torch.ones(1, 768)
|
||||
packed = adapter.pack(x, timestep, context, y=y, guidance=torch.tensor([1.0]))
|
||||
assert packed.latents.ndim == 3
|
||||
assert packed.pooled_embeds[0] is y
|
||||
assert packed.guidance_scale == 1.0
|
||||
out = adapter.unpack(packed.latents, packed, x)
|
||||
assert out.shape == x.shape
|
||||
assert torch.equal(out, x)
|
||||
|
||||
default = adapter.pack(x, timestep, context, y=y)
|
||||
assert default.guidance_scale == 3.5
|
||||
@@ -0,0 +1,85 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Process-wide SGLD worker ownership for ComfyUI loaders."""
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
from sglang.multimodal_gen.apps.ComfyUI_SGLDiffusion.core.generator import (
|
||||
SGLDiffusionGenerator,
|
||||
)
|
||||
|
||||
|
||||
def test_shared_is_process_singleton() -> None:
|
||||
SGLDiffusionGenerator.reset_shared()
|
||||
assert SGLDiffusionGenerator.shared() is SGLDiffusionGenerator.shared()
|
||||
SGLDiffusionGenerator.reset_shared()
|
||||
|
||||
|
||||
def test_reuse_requires_live_worker() -> None:
|
||||
runtime = SGLDiffusionGenerator()
|
||||
options = {"model_path": "z.safetensors"}
|
||||
runtime.last_options = options
|
||||
runtime.generator = object()
|
||||
runtime._patcher = object()
|
||||
runtime.executor = object()
|
||||
runtime._is_live = lambda: False
|
||||
assert runtime._can_reuse(options) is False
|
||||
runtime._is_live = lambda: True
|
||||
assert runtime._can_reuse(options) is True
|
||||
assert runtime._can_reuse({"model_path": "h3.safetensors"}) is False
|
||||
|
||||
|
||||
def test_ensure_rebuilds_when_another_model_owns_the_worker() -> None:
|
||||
runtime = SGLDiffusionGenerator()
|
||||
stale = object()
|
||||
fresh = object()
|
||||
loads = []
|
||||
executor = SimpleNamespace(
|
||||
generator=stale,
|
||||
_sgld_reload={
|
||||
"model_path": "z.safetensors",
|
||||
"model_options": {},
|
||||
"sgld_options": {},
|
||||
},
|
||||
_lora_input=None,
|
||||
)
|
||||
|
||||
def fake_load(**kwargs):
|
||||
loads.append(kwargs)
|
||||
runtime.generator = fresh
|
||||
return "patcher"
|
||||
|
||||
runtime.load_model = fake_load
|
||||
runtime.ensure_executor(executor)
|
||||
assert loads == [executor._sgld_reload]
|
||||
assert executor.generator is fresh
|
||||
|
||||
|
||||
def test_ensure_is_noop_when_executor_still_owns_live_worker() -> None:
|
||||
runtime = SGLDiffusionGenerator()
|
||||
gen = object()
|
||||
runtime.generator = gen
|
||||
runtime._is_live = lambda: True
|
||||
executor = SimpleNamespace(generator=gen, _sgld_reload={"model_path": "z"})
|
||||
runtime.load_model = lambda **kwargs: (_ for _ in ()).throw(
|
||||
AssertionError("should not reload")
|
||||
)
|
||||
runtime.ensure_executor(executor)
|
||||
assert executor.generator is gen
|
||||
|
||||
|
||||
def test_kill_generator_only_touches_owned_workers() -> None:
|
||||
runtime = SGLDiffusionGenerator()
|
||||
owned = SimpleNamespace(alive=True, terminated=False, killed=False, pid=9)
|
||||
|
||||
def terminate():
|
||||
owned.terminated = True
|
||||
owned.alive = False
|
||||
|
||||
owned.is_alive = lambda: owned.alive
|
||||
owned.terminate = terminate
|
||||
owned.join = lambda timeout=None: None
|
||||
owned.kill = lambda: setattr(owned, "killed", True)
|
||||
runtime.generator = SimpleNamespace(local_scheduler_process=[owned])
|
||||
runtime.kill_generator()
|
||||
assert owned.terminated is True
|
||||
assert owned.killed is False
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,205 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Tests for the comfyui_mode profile of native pipelines.
|
||||
|
||||
The existing ComfyUI tests under apps/ComfyUI_SGLDiffusion/test need a real
|
||||
multi-GB checkpoint and two GPUs, so nothing here is covered in CI. These build
|
||||
a tiny ComfyUI-format checkpoint instead, which is enough to exercise module
|
||||
trimming, single-file loading, weight conversion, and stage construction.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from safetensors.torch import save_file
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.flux import FluxConfig
|
||||
from sglang.multimodal_gen.registry import get_pipeline_class
|
||||
from sglang.multimodal_gen.runtime.distributed.parallel_state import (
|
||||
maybe_init_distributed_environment_and_model_parallel,
|
||||
model_parallel_is_initialized,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.loader.comfyui_checkpoints import (
|
||||
get_comfyui_checkpoint_spec,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
from sglang.multimodal_gen.runtime.server_args.server_args import set_global_server_args
|
||||
from sglang.multimodal_gen.test.single_test_file.component_accuracy.utils import (
|
||||
ensure_distributed_env_defaults,
|
||||
)
|
||||
|
||||
HEADS, HEAD_DIM, LAYERS, SINGLE_LAYERS = 4, 16, 2, 2
|
||||
|
||||
|
||||
def _ensure_single_process_parallel_runtime() -> None:
|
||||
if model_parallel_is_initialized():
|
||||
return
|
||||
ensure_distributed_env_defaults()
|
||||
maybe_init_distributed_environment_and_model_parallel(tp_size=1, sp_size=1)
|
||||
|
||||
|
||||
def _shrink(arch) -> None:
|
||||
arch.num_attention_heads = HEADS
|
||||
arch.attention_head_dim = HEAD_DIM
|
||||
arch.num_layers = LAYERS
|
||||
arch.num_single_layers = SINGLE_LAYERS
|
||||
arch.guidance_embeds = True
|
||||
|
||||
|
||||
def _write_comfyui_flux_checkpoint(path: str, config: FluxConfig) -> dict:
|
||||
"""Write a checkpoint using ComfyUI's parameter names and fused layouts."""
|
||||
arch = config.arch_config
|
||||
hidden = arch.num_attention_heads * arch.attention_head_dim
|
||||
mlp_hidden = int(hidden * getattr(arch, "mlp_ratio", 4.0))
|
||||
qkv = 3 * hidden
|
||||
out_channels = arch.out_channels or arch.in_channels
|
||||
patch = getattr(arch, "patch_size", 1)
|
||||
|
||||
sd: dict[str, torch.Tensor] = {}
|
||||
|
||||
def put(name: str, *shape: int) -> None:
|
||||
sd[name] = torch.randn(*shape, dtype=torch.bfloat16)
|
||||
|
||||
for b in range(arch.num_layers):
|
||||
for attn in ("img_attn", "txt_attn"):
|
||||
put(f"double_blocks.{b}.{attn}.qkv.weight", qkv, hidden)
|
||||
put(f"double_blocks.{b}.{attn}.qkv.bias", qkv)
|
||||
put(f"double_blocks.{b}.{attn}.proj.weight", hidden, hidden)
|
||||
put(f"double_blocks.{b}.{attn}.proj.bias", hidden)
|
||||
put(
|
||||
f"double_blocks.{b}.{attn}.norm.query_norm.scale",
|
||||
arch.attention_head_dim,
|
||||
)
|
||||
put(
|
||||
f"double_blocks.{b}.{attn}.norm.key_norm.scale", arch.attention_head_dim
|
||||
)
|
||||
for mlp in ("img_mlp", "txt_mlp"):
|
||||
put(f"double_blocks.{b}.{mlp}.0.weight", mlp_hidden, hidden)
|
||||
put(f"double_blocks.{b}.{mlp}.0.bias", mlp_hidden)
|
||||
put(f"double_blocks.{b}.{mlp}.2.weight", hidden, mlp_hidden)
|
||||
put(f"double_blocks.{b}.{mlp}.2.bias", hidden)
|
||||
for mod in ("img_mod", "txt_mod"):
|
||||
put(f"double_blocks.{b}.{mod}.lin.weight", 6 * hidden, hidden)
|
||||
put(f"double_blocks.{b}.{mod}.lin.bias", 6 * hidden)
|
||||
|
||||
for b in range(arch.num_single_layers):
|
||||
put(f"single_blocks.{b}.linear1.weight", qkv + mlp_hidden, hidden)
|
||||
put(f"single_blocks.{b}.linear1.bias", qkv + mlp_hidden)
|
||||
put(f"single_blocks.{b}.linear2.weight", hidden, hidden + mlp_hidden)
|
||||
put(f"single_blocks.{b}.linear2.bias", hidden)
|
||||
put(f"single_blocks.{b}.norm.query_norm.scale", arch.attention_head_dim)
|
||||
put(f"single_blocks.{b}.norm.key_norm.scale", arch.attention_head_dim)
|
||||
put(f"single_blocks.{b}.modulation.lin.weight", 3 * hidden, hidden)
|
||||
put(f"single_blocks.{b}.modulation.lin.bias", 3 * hidden)
|
||||
|
||||
for stem, in_dim in (
|
||||
("time_in", 256),
|
||||
("vector_in", arch.pooled_projection_dim),
|
||||
("guidance_in", 256),
|
||||
):
|
||||
put(f"{stem}.in_layer.weight", hidden, in_dim)
|
||||
put(f"{stem}.in_layer.bias", hidden)
|
||||
put(f"{stem}.out_layer.weight", hidden, hidden)
|
||||
put(f"{stem}.out_layer.bias", hidden)
|
||||
|
||||
put("txt_in.weight", hidden, arch.joint_attention_dim)
|
||||
put("txt_in.bias", hidden)
|
||||
put("img_in.weight", hidden, arch.in_channels)
|
||||
put("img_in.bias", hidden)
|
||||
put("final_layer.linear.weight", patch * patch * out_channels, hidden)
|
||||
put("final_layer.linear.bias", patch * patch * out_channels)
|
||||
put("final_layer.adaLN_modulation.1.weight", 2 * hidden, hidden)
|
||||
put("final_layer.adaLN_modulation.1.bias", 2 * hidden)
|
||||
|
||||
save_file(sd, path)
|
||||
return sd
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def comfyui_flux_pipeline():
|
||||
if not torch.cuda.is_available():
|
||||
pytest.skip("requires CUDA")
|
||||
|
||||
tmpdir = tempfile.mkdtemp(prefix="comfyui_flux_")
|
||||
checkpoint = os.path.join(tmpdir, "flux_comfyui.safetensors")
|
||||
|
||||
# The file has to exist before ServerArgs resolves the single-file path.
|
||||
seed_config = FluxConfig()
|
||||
_shrink(seed_config.arch_config)
|
||||
state_dict = _write_comfyui_flux_checkpoint(checkpoint, seed_config)
|
||||
|
||||
server_args = ServerArgs.from_kwargs(
|
||||
model_path=checkpoint,
|
||||
pipeline_class_name="FluxPipeline",
|
||||
comfyui_mode=True,
|
||||
num_gpus=1,
|
||||
)
|
||||
_shrink(server_args.pipeline_config.dit_config.arch_config)
|
||||
set_global_server_args(server_args)
|
||||
_ensure_single_process_parallel_runtime()
|
||||
|
||||
pipeline = get_pipeline_class("FluxPipeline")(
|
||||
model_path=checkpoint, server_args=server_args
|
||||
)
|
||||
return pipeline, state_dict
|
||||
|
||||
|
||||
def test_specs_cover_every_comfyui_supported_pipeline():
|
||||
for pipeline_name in (
|
||||
"FluxPipeline",
|
||||
"ZImagePipeline",
|
||||
"QwenImagePipeline",
|
||||
"QwenImageEditPlusPipeline",
|
||||
):
|
||||
spec = get_comfyui_checkpoint_spec(pipeline_name)
|
||||
assert spec is not None, f"{pipeline_name} has no ComfyUI checkpoint spec"
|
||||
assert get_pipeline_class(pipeline_name) is not None
|
||||
assert get_pipeline_class(pipeline_name).pipeline_config_cls is not None
|
||||
|
||||
|
||||
def test_comfyui_mode_trims_pipeline_to_a_dit_forward_service(comfyui_flux_pipeline):
|
||||
pipeline, _ = comfyui_flux_pipeline
|
||||
|
||||
assert pipeline.required_config_modules == ["transformer", "scheduler"]
|
||||
assert (
|
||||
type(pipeline.get_module("scheduler")).__name__ == "ComfyUIPassThroughScheduler"
|
||||
)
|
||||
assert list(pipeline._stage_name_mapping) == [
|
||||
"ComfyUILatentPreparationStage",
|
||||
"DenoisingStage",
|
||||
]
|
||||
|
||||
|
||||
def test_comfyui_checkpoint_fully_populates_the_transformer(comfyui_flux_pipeline):
|
||||
pipeline, _ = comfyui_flux_pipeline
|
||||
transformer = pipeline.get_module("transformer")
|
||||
|
||||
unloaded = [name for name, p in transformer.named_parameters() if p.is_meta]
|
||||
assert not unloaded, f"parameters never received checkpoint weights: {unloaded[:5]}"
|
||||
|
||||
|
||||
def test_fused_qkv_is_split_into_separate_projections(comfyui_flux_pipeline):
|
||||
pipeline, state_dict = comfyui_flux_pipeline
|
||||
params = dict(pipeline.get_module("transformer").named_parameters())
|
||||
device = next(iter(params.values())).device
|
||||
hidden = HEADS * HEAD_DIM
|
||||
|
||||
fused = state_dict["double_blocks.0.img_attn.qkv.weight"].to(device, torch.bfloat16)
|
||||
for offset, proj in enumerate(("to_q", "to_k", "to_v")):
|
||||
expected = fused[offset * hidden : (offset + 1) * hidden]
|
||||
actual = params[f"transformer_blocks.0.attn.{proj}.weight"]
|
||||
assert torch.equal(actual, expected), f"{proj} does not match the fused slice"
|
||||
|
||||
|
||||
def test_adaln_modulation_scale_and_shift_are_swapped(comfyui_flux_pipeline):
|
||||
pipeline, state_dict = comfyui_flux_pipeline
|
||||
params = dict(pipeline.get_module("transformer").named_parameters())
|
||||
actual = params["norm_out.linear.weight"]
|
||||
|
||||
source = state_dict["final_layer.adaLN_modulation.1.weight"].to(
|
||||
actual.device, torch.bfloat16
|
||||
)
|
||||
half = source.shape[0] // 2
|
||||
# ComfyUI emits [shift, scale]; AdaLayerNormContinuous expects [scale, shift].
|
||||
assert torch.equal(actual, torch.cat([source[half:], source[:half]], dim=0))
|
||||
@@ -0,0 +1,229 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.comfyui_mode import (
|
||||
bind_comfyui_session,
|
||||
get_run_state,
|
||||
initialize_comfyui_pipeline,
|
||||
release_comfyui_session,
|
||||
release_run_state,
|
||||
set_run_state,
|
||||
)
|
||||
|
||||
|
||||
class _Req:
|
||||
def __init__(self):
|
||||
self.extra = {}
|
||||
self.prompt_embeds = []
|
||||
self.prompt_seq_lens = None
|
||||
self.pooled_embeds = []
|
||||
|
||||
|
||||
def test_latent_prep_verify_input_restores_cached_embeds() -> None:
|
||||
"""Later ComfyUI steps omit embeds; verify_input must restore before checks."""
|
||||
from types import SimpleNamespace
|
||||
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.comfyui_latent_preparation import (
|
||||
ComfyUILatentPreparationStage,
|
||||
)
|
||||
|
||||
sid = "run-verify"
|
||||
embeds = [torch.ones(2, 4)]
|
||||
first = _Req()
|
||||
first.extra["comfyui_session_id"] = sid
|
||||
first.prompt_embeds = embeds
|
||||
first.prompt_seq_lens = [[2]]
|
||||
bind_comfyui_session(first)
|
||||
|
||||
batch = SimpleNamespace(
|
||||
extra={"comfyui_session_id": sid},
|
||||
prompt_embeds=[],
|
||||
prompt=" ",
|
||||
num_outputs_per_prompt=1,
|
||||
generator=torch.Generator("cpu"),
|
||||
num_frames=1,
|
||||
height=64,
|
||||
width=64,
|
||||
latents=None,
|
||||
prompt_seq_lens=None,
|
||||
pooled_embeds=None,
|
||||
negative_prompt_embeds=None,
|
||||
negative_prompt_seq_lens=None,
|
||||
neg_pooled_embeds=None,
|
||||
image_latent=None,
|
||||
vae_image_sizes=None,
|
||||
prompt_attention_mask=None,
|
||||
negative_attention_mask=None,
|
||||
prompt_embeds_mask=None,
|
||||
negative_prompt_embeds_mask=None,
|
||||
)
|
||||
stage = ComfyUILatentPreparationStage(scheduler=None, transformer=None)
|
||||
result = stage.verify_input(batch, server_args=None)
|
||||
assert result.is_valid()
|
||||
assert torch.equal(batch.prompt_embeds[0], embeds[0])
|
||||
release_comfyui_session(sid)
|
||||
|
||||
|
||||
def test_session_restores_conditioning_on_later_steps() -> None:
|
||||
sid = "run-1"
|
||||
first = _Req()
|
||||
first.extra["comfyui_session_id"] = sid
|
||||
first.prompt_embeds = [torch.ones(4, 8)]
|
||||
first.prompt_seq_lens = [[4]]
|
||||
bind_comfyui_session(first)
|
||||
|
||||
second = _Req()
|
||||
second.extra["comfyui_session_id"] = sid
|
||||
bind_comfyui_session(second)
|
||||
|
||||
assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0])
|
||||
assert second.prompt_seq_lens == [[4]]
|
||||
release_comfyui_session(sid)
|
||||
|
||||
third = _Req()
|
||||
third.extra["comfyui_session_id"] = sid
|
||||
bind_comfyui_session(third)
|
||||
assert third.prompt_embeds == []
|
||||
|
||||
|
||||
def test_release_run_state_keeps_conditioning():
|
||||
sid = "run-keep-cond"
|
||||
first = _Req()
|
||||
first.extra = {"comfyui_session_id": sid, "model_payload": {"audio_scale": 0.5}}
|
||||
first.prompt_embeds = [torch.ones(2, 4)]
|
||||
bind_comfyui_session(first)
|
||||
set_run_state(first, {"branch": "alive"})
|
||||
release_run_state(sid)
|
||||
assert get_run_state(first) is None
|
||||
second = _Req()
|
||||
second.extra = {"comfyui_session_id": sid}
|
||||
bind_comfyui_session(second)
|
||||
assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0])
|
||||
assert second.extra["model_payload"]["audio_scale"] == 0.5
|
||||
release_comfyui_session(sid)
|
||||
|
||||
|
||||
def test_session_restores_extra_keys_on_later_steps() -> None:
|
||||
sid = "run-extra"
|
||||
payload = {"audio_scale": 0.5}
|
||||
sigmas = torch.tensor([1.0, 0.5, 0.0])
|
||||
first = _Req()
|
||||
first.extra = {
|
||||
"comfyui_session_id": sid,
|
||||
"model_payload": payload,
|
||||
"sample_sigmas": sigmas,
|
||||
}
|
||||
first.prompt_embeds = [torch.ones(3, 4)]
|
||||
bind_comfyui_session(first)
|
||||
|
||||
second = _Req()
|
||||
second.extra = {"comfyui_session_id": sid}
|
||||
bind_comfyui_session(second)
|
||||
assert second.extra["model_payload"] is payload
|
||||
assert torch.equal(second.extra["sample_sigmas"], sigmas)
|
||||
assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0])
|
||||
release_comfyui_session(sid)
|
||||
|
||||
|
||||
def test_new_run_id_evicts_previous_run_for_same_executor() -> None:
|
||||
first = _Req()
|
||||
first.extra = {"comfyui_session_id": "exec1:1", "h3_layout": {"signature": (1, 2)}}
|
||||
first.prompt_embeds = [torch.ones(2, 4)]
|
||||
bind_comfyui_session(first)
|
||||
set_run_state(first, {"branch": "old"})
|
||||
|
||||
second = _Req()
|
||||
second.extra = {"comfyui_session_id": "exec1:2"}
|
||||
bind_comfyui_session(second)
|
||||
assert "h3_layout" not in second.extra
|
||||
assert second.prompt_embeds == []
|
||||
old = _Req()
|
||||
old.extra = {"comfyui_session_id": "exec1:1"}
|
||||
assert get_run_state(old) is None
|
||||
release_comfyui_session("exec1:2")
|
||||
|
||||
|
||||
def test_fingerprint_mismatch_does_not_restore_extras() -> None:
|
||||
sid = "exec2:1"
|
||||
first = _Req()
|
||||
first.extra = {
|
||||
"comfyui_session_id": sid,
|
||||
"comfyui_cache_fp": {"spatial": (5, 15, 27)},
|
||||
"h3_layout": {"signature": (8, 5, 15, 27, 4)},
|
||||
}
|
||||
first.prompt_embeds = [torch.ones(2, 4)]
|
||||
bind_comfyui_session(first)
|
||||
|
||||
second = _Req()
|
||||
second.extra = {
|
||||
"comfyui_session_id": sid,
|
||||
"comfyui_cache_fp": {"spatial": (5, 30, 54)},
|
||||
}
|
||||
bind_comfyui_session(second)
|
||||
assert "h3_layout" not in second.extra
|
||||
assert torch.equal(second.prompt_embeds[0], first.prompt_embeds[0])
|
||||
release_comfyui_session(sid)
|
||||
|
||||
|
||||
def test_cond_keys_keep_positive_and_negative_apart() -> None:
|
||||
sid = "exec3:1"
|
||||
pos = _Req()
|
||||
pos.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "pos"}
|
||||
pos.prompt_embeds = [torch.ones(2, 4)]
|
||||
bind_comfyui_session(pos)
|
||||
neg = _Req()
|
||||
neg.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "neg"}
|
||||
neg.prompt_embeds = [torch.zeros(2, 4)]
|
||||
bind_comfyui_session(neg)
|
||||
|
||||
later_pos = _Req()
|
||||
later_pos.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "pos"}
|
||||
bind_comfyui_session(later_pos)
|
||||
later_neg = _Req()
|
||||
later_neg.extra = {"comfyui_session_id": sid, "comfyui_cond_key": "neg"}
|
||||
bind_comfyui_session(later_neg)
|
||||
assert torch.equal(later_pos.prompt_embeds[0], pos.prompt_embeds[0])
|
||||
assert torch.equal(later_neg.prompt_embeds[0], neg.prompt_embeds[0])
|
||||
release_comfyui_session(sid)
|
||||
|
||||
|
||||
class _FakePipeline:
|
||||
def __init__(self):
|
||||
self.modules = {}
|
||||
|
||||
|
||||
def _init_vae_geometry(pipeline_config):
|
||||
from types import SimpleNamespace
|
||||
|
||||
initialize_comfyui_pipeline(
|
||||
_FakePipeline(),
|
||||
SimpleNamespace(pipeline_config=pipeline_config, comfyui_mode=True),
|
||||
)
|
||||
return pipeline_config.vae_config.arch_config
|
||||
|
||||
|
||||
def test_comfyui_mode_derives_flux_vae_scale_factor() -> None:
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.flux import FluxPipelineConfig
|
||||
|
||||
arch = _init_vae_geometry(FluxPipelineConfig())
|
||||
assert arch.vae_scale_factor == 8
|
||||
|
||||
|
||||
def test_comfyui_mode_derives_qwen_vae_scale_factor() -> None:
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.qwen_image import (
|
||||
QwenImagePipelineConfig,
|
||||
)
|
||||
|
||||
arch = _init_vae_geometry(QwenImagePipelineConfig())
|
||||
assert arch.vae_scale_factor == 8
|
||||
|
||||
|
||||
def test_comfyui_mode_h3_vae_post_init_without_latent_stats() -> None:
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.minimax_h3 import (
|
||||
MiniMaxH3PipelineConfig,
|
||||
)
|
||||
|
||||
arch = _init_vae_geometry(MiniMaxH3PipelineConfig())
|
||||
assert arch.latents_mean is None
|
||||
assert arch.latents_std is None
|
||||
@@ -92,6 +92,7 @@ class TestComponentLoaderIdentity(unittest.TestCase):
|
||||
component_paths={},
|
||||
component_direct_gpu_weight_loading=set(),
|
||||
resolve_component_attention_backend=lambda *_names: (None, None),
|
||||
comfyui_mode=False,
|
||||
)
|
||||
|
||||
with patch.object(
|
||||
|
||||
@@ -443,6 +443,46 @@ class TestRankLocalSafetensorsRead(unittest.TestCase):
|
||||
rank_local_checkpoint.tp_local_shape(sources, 0, 2), (4, 4)
|
||||
)
|
||||
|
||||
def test_reads_tp_local_fused_merged_column_per_logical_matrix(self):
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
file_path = str(Path(temp_dir) / "model.safetensors")
|
||||
q = torch.arange(16, dtype=torch.bfloat16).reshape(4, 4)
|
||||
k = torch.arange(16, 32, dtype=torch.bfloat16).reshape(4, 4)
|
||||
v = torch.arange(32, 48, dtype=torch.bfloat16).reshape(4, 4)
|
||||
fused = torch.cat((q, k, v), dim=0)
|
||||
save_file({"qkv": fused}, file_path)
|
||||
sources = [self._source(file_path, "qkv", (12, 4))]
|
||||
|
||||
with safe_open(file_path, framework="pt", device="cpu") as handle:
|
||||
naive = rank_local_checkpoint.read_tp_local_tensor(
|
||||
sources,
|
||||
{file_path: handle},
|
||||
shard_dim=0,
|
||||
tp_rank=0,
|
||||
tp_size=2,
|
||||
)
|
||||
rank0 = rank_local_checkpoint.read_tp_local_tensor(
|
||||
sources,
|
||||
{file_path: handle},
|
||||
shard_dim=0,
|
||||
tp_rank=0,
|
||||
tp_size=2,
|
||||
output_sizes=[4, 4, 4],
|
||||
)
|
||||
rank1 = rank_local_checkpoint.read_tp_local_tensor(
|
||||
sources,
|
||||
{file_path: handle},
|
||||
shard_dim=0,
|
||||
tp_rank=1,
|
||||
tp_size=2,
|
||||
output_sizes=[4, 4, 4],
|
||||
)
|
||||
|
||||
torch.testing.assert_close(naive, fused[:6])
|
||||
torch.testing.assert_close(rank0, torch.cat((q[:2], k[:2], v[:2])))
|
||||
torch.testing.assert_close(rank1, torch.cat((q[2:], k[2:], v[2:])))
|
||||
self.assertFalse(torch.equal(rank0, naive))
|
||||
|
||||
def test_reads_tp_local_merged_row_slice(self):
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
file_path = str(Path(temp_dir) / "model.safetensors")
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import pickle
|
||||
from dataclasses import dataclass
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.runtime.distributed.ipc_cuda import (
|
||||
CudaIpcRef,
|
||||
attach_cuda_tensors,
|
||||
detach_cuda_tensors,
|
||||
materialize_cuda_refs,
|
||||
spill_cuda_tensors,
|
||||
)
|
||||
|
||||
|
||||
def _require_cuda() -> None:
|
||||
if not torch.cuda.is_available():
|
||||
pytest.skip("CUDA is required for CUDA IPC tests")
|
||||
|
||||
|
||||
def test_pickle_copies_cuda_tensor_through_host() -> None:
|
||||
_require_cuda()
|
||||
tensor = torch.ones(16, 90, 160, device="cuda", dtype=torch.bfloat16)
|
||||
assert len(pickle.dumps(tensor)) > tensor.nbytes
|
||||
|
||||
|
||||
def test_ipc_handle_pickle_is_much_smaller_than_tensor() -> None:
|
||||
_require_cuda()
|
||||
tensor = torch.ones(16, 90, 160, device="cuda", dtype=torch.bfloat16)
|
||||
ref = CudaIpcRef.from_tensor(tensor)
|
||||
assert len(pickle.dumps(ref)) < 2048
|
||||
rebuilt = ref.materialize()
|
||||
assert rebuilt.device.type == "cuda"
|
||||
assert rebuilt.shape == tensor.shape
|
||||
assert rebuilt.dtype == tensor.dtype
|
||||
assert torch.equal(rebuilt, tensor)
|
||||
|
||||
|
||||
def test_spill_does_not_mutate_caller_req_tensors() -> None:
|
||||
_require_cuda()
|
||||
|
||||
@dataclass
|
||||
class _Holder:
|
||||
latents: torch.Tensor
|
||||
prompt_embeds: list
|
||||
|
||||
latents = torch.arange(24, device="cuda", dtype=torch.float32).reshape(2, 3, 4)
|
||||
embeds = [torch.ones(3, 5, device="cuda", dtype=torch.float32)]
|
||||
holder = _Holder(latents=latents, prompt_embeds=embeds)
|
||||
|
||||
spilled = spill_cuda_tensors(holder)
|
||||
assert isinstance(spilled.latents, CudaIpcRef)
|
||||
assert isinstance(spilled.prompt_embeds[0], CudaIpcRef)
|
||||
assert holder.latents is latents
|
||||
assert holder.prompt_embeds[0] is embeds[0]
|
||||
|
||||
restored = materialize_cuda_refs(spilled)
|
||||
assert torch.equal(restored.latents, latents)
|
||||
assert torch.equal(restored.prompt_embeds[0], embeds[0])
|
||||
|
||||
|
||||
def test_detach_attach_keeps_non_cuda_fields() -> None:
|
||||
_require_cuda()
|
||||
|
||||
@dataclass
|
||||
class _Holder:
|
||||
extra: dict
|
||||
vae_image_sizes: list
|
||||
image_embeds: list
|
||||
latents: torch.Tensor
|
||||
prompt_embeds: list
|
||||
|
||||
latents = torch.arange(24, device="cuda", dtype=torch.float32).reshape(2, 3, 4)
|
||||
embeds = [torch.ones(2, 8, device="cuda", dtype=torch.float32)]
|
||||
holder = _Holder(
|
||||
extra={"comfyui_session_id": "run-1"},
|
||||
vae_image_sizes=[(64, 48)],
|
||||
image_embeds=[],
|
||||
latents=latents,
|
||||
prompt_embeds=embeds,
|
||||
)
|
||||
|
||||
skeleton, tensors = detach_cuda_tensors(holder)
|
||||
assert holder.latents is latents
|
||||
assert skeleton.extra["comfyui_session_id"] == "run-1"
|
||||
assert skeleton.vae_image_sizes == [(64, 48)]
|
||||
assert skeleton.image_embeds == []
|
||||
assert skeleton.latents is None
|
||||
|
||||
restored = attach_cuda_tensors(skeleton, tensors)
|
||||
assert restored.extra["comfyui_session_id"] == "run-1"
|
||||
assert restored.vae_image_sizes == [(64, 48)]
|
||||
assert restored.image_embeds == []
|
||||
assert restored.latents.device.type == "cuda"
|
||||
assert torch.equal(restored.latents, latents)
|
||||
assert torch.equal(restored.prompt_embeds[0], embeds[0])
|
||||
|
||||
|
||||
def test_cross_process_ipc_roundtrip() -> None:
|
||||
_require_cuda()
|
||||
ctx = torch.multiprocessing.get_context("spawn")
|
||||
tensor = torch.arange(64, device="cuda", dtype=torch.float32).reshape(4, 16)
|
||||
ref = CudaIpcRef.from_tensor(tensor)
|
||||
queue = ctx.Queue()
|
||||
proc = ctx.Process(target=_cross_process_child, args=(queue, pickle.dumps(ref)))
|
||||
proc.start()
|
||||
try:
|
||||
shape, dtype, first, last = queue.get(timeout=30)
|
||||
finally:
|
||||
proc.join(timeout=30)
|
||||
if proc.is_alive():
|
||||
proc.kill()
|
||||
proc.join()
|
||||
assert proc.exitcode == 0
|
||||
assert shape == (4, 16)
|
||||
assert dtype == "torch.float32"
|
||||
assert first == 0.0
|
||||
assert last == 63.0
|
||||
|
||||
|
||||
def _cross_process_child(queue, payload: bytes) -> None:
|
||||
ref = pickle.loads(payload)
|
||||
rebuilt = ref.materialize()
|
||||
queue.put(
|
||||
(
|
||||
tuple(rebuilt.shape),
|
||||
str(rebuilt.dtype),
|
||||
float(rebuilt.reshape(-1)[0]),
|
||||
float(rebuilt.reshape(-1)[-1]),
|
||||
)
|
||||
)
|
||||
Reference in New Issue
Block a user