[diffusion] comfyui: add a minimax-h3 node and a generic extra-fields passthrough (#35352)
This commit is contained in:
@@ -574,6 +574,39 @@ After implementation, **you must verify that the generated output is not noise**
|
|||||||
2. Running the Diffusers pipeline and SGLang pipeline side-by-side with the same seed
|
2. Running the Diffusers pipeline and SGLang pipeline side-by-side with the same seed
|
||||||
3. Checking each stage's output shape and value range independently
|
3. Checking each stage's output shape and value range independently
|
||||||
|
|
||||||
|
### Step 10: Decide the ComfyUI Route (Optional)
|
||||||
|
|
||||||
|
A model is reachable from ComfyUI two ways. Pick one deliberately — the wrong
|
||||||
|
choice costs several hundred lines of weight-mapping code that buys nothing.
|
||||||
|
|
||||||
|
**Server route.** ComfyUI sends an HTTP request and SGLang runs the whole
|
||||||
|
pipeline. Choose this when the model needs conditioning ComfyUI cannot supply
|
||||||
|
(audio, reference materials, task routing), produces more than one modality,
|
||||||
|
or has its own request contract.
|
||||||
|
|
||||||
|
Cost: nothing, if the request fits the existing `generate_image` /
|
||||||
|
`generate_video` fields. If the model has extra request fields, pass them
|
||||||
|
through `extra_fields` — the request schemas accept unknown keys, so the
|
||||||
|
client in `apps/ComfyUI_SGLDiffusion/core/server_api.py` does **not** need a
|
||||||
|
per-model change. Add a node in `nodes.py` only when the inputs are worth
|
||||||
|
surfacing as ComfyUI widgets. `SGLDiffusionGenerateH3` is the worked example.
|
||||||
|
|
||||||
|
**Executor route.** ComfyUI's KSampler drives the denoise loop and SGLang
|
||||||
|
replaces the DiT forward, using ComfyUI's own text encoders and VAE. Choose
|
||||||
|
this only when the model denoises a single latent tensor that ComfyUI already
|
||||||
|
knows how to build and decode.
|
||||||
|
|
||||||
|
Cost, per model: a `runtime/pipelines/comfyui_<model>_pipeline.py` that maps
|
||||||
|
ComfyUI's single-file checkpoint layout onto the native module tree (350-690
|
||||||
|
lines in the existing three), an executor in
|
||||||
|
`apps/ComfyUI_SGLDiffusion/executors/` that adapts latent layout and
|
||||||
|
conditioning to `Req`, and entries in both dicts in `core/generator.py`.
|
||||||
|
|
||||||
|
The deciding question is not model size or modality — it is whether ComfyUI's
|
||||||
|
sampler can drive the model's loop unchanged. If reproducing the conditioning
|
||||||
|
inside ComfyUI would duplicate stages the server already runs, take the server
|
||||||
|
route.
|
||||||
|
|
||||||
## Reference Implementations
|
## Reference Implementations
|
||||||
|
|
||||||
### Hybrid Style (recommended for most new models)
|
### Hybrid Style (recommended for most new models)
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ The plugin supports two modes of operation: **Server Mode** (via HTTP API) and *
|
|||||||
- **Z-Image**: High-speed image generation models (e.g., `Z-Image-Turbo`)
|
- **Z-Image**: High-speed image generation models (e.g., `Z-Image-Turbo`)
|
||||||
- **FLUX**: State-of-the-art text-to-image models (e.g., `FLUX.1-dev`)
|
- **FLUX**: State-of-the-art text-to-image models (e.g., `FLUX.1-dev`)
|
||||||
- **Qwen-Image**: Multi-modal image generation models (e.g., `Qwen-Image`,`Qwen-Image-2512`). *Note: Image editing support is currently experimental and may have some issues.*
|
- **Qwen-Image**: Multi-modal image generation models (e.g., `Qwen-Image`,`Qwen-Image-2512`). *Note: Image editing support is currently experimental and may have some issues.*
|
||||||
|
- **MiniMax-H3**: Joint video-and-audio generation, server mode only (`SGLDiffusion Generate H3`)
|
||||||
|
|
||||||
### Mode 1: Server Mode (HTTP API)
|
### Mode 1: Server Mode (HTTP API)
|
||||||
Connect to a standalone SGLang Diffusion server.
|
Connect to a standalone SGLang Diffusion server.
|
||||||
@@ -35,6 +36,35 @@ Leverage SGLang's high-performance sampling directly within ComfyUI while using
|
|||||||
3. **Sample**: Connect the loaded model to standard ComfyUI samplers. SGLang will handle the sampling process efficiently.
|
3. **Sample**: Connect the loaded model to standard ComfyUI samplers. SGLang will handle the sampling process efficiently.
|
||||||
4. **LoRA Support**: Use the `SGLDiffusion LoRA Loader` for native LoRA integration.
|
4. **LoRA Support**: Use the `SGLDiffusion LoRA Loader` for native LoRA integration.
|
||||||
|
|
||||||
|
## Adding a Model
|
||||||
|
|
||||||
|
Pick the mode before writing code; the wrong one costs several hundred lines
|
||||||
|
of weight mapping that buys nothing.
|
||||||
|
|
||||||
|
Take **Server Mode** when the model needs conditioning ComfyUI cannot supply
|
||||||
|
(audio, reference materials, task routing), emits more than one modality, or
|
||||||
|
has its own request contract. Reproducing that inside ComfyUI would duplicate
|
||||||
|
stages the server already runs.
|
||||||
|
|
||||||
|
- If the request fits the existing `generate_image` / `generate_video` fields,
|
||||||
|
there is nothing to write — point the existing nodes at the server.
|
||||||
|
- If the model has extra request fields, pass them via `extra_fields`. The
|
||||||
|
request schemas accept unknown keys, so `core/server_api.py` needs no
|
||||||
|
per-model change.
|
||||||
|
- Add a node in `nodes.py` only to surface those inputs as ComfyUI widgets.
|
||||||
|
`SGLDiffusionGenerateH3` is the worked example.
|
||||||
|
|
||||||
|
Take **Integrated Mode** only when the model denoises a single latent tensor
|
||||||
|
that ComfyUI already knows how to build and decode, so its KSampler can drive
|
||||||
|
the loop unchanged. Each model then needs:
|
||||||
|
|
||||||
|
- `runtime/pipelines/comfyui_<model>_pipeline.py` mapping ComfyUI's
|
||||||
|
single-file checkpoint layout onto the native module tree (350-690 lines in
|
||||||
|
the existing three)
|
||||||
|
- an executor in `executors/` adapting latent layout and conditioning to `Req`
|
||||||
|
- entries in both `pipeline_class_dict` and `executor_class_dict` in
|
||||||
|
`core/generator.py`
|
||||||
|
|
||||||
## Example Workflows
|
## Example Workflows
|
||||||
|
|
||||||
Reference workflow files are provided in the `workflows/` directory:
|
Reference workflow files are provided in the `workflows/` directory:
|
||||||
|
|||||||
@@ -219,6 +219,7 @@ class SGLDiffusionServerAPI:
|
|||||||
generator_device: Optional[str] = "cuda",
|
generator_device: Optional[str] = "cuda",
|
||||||
input_reference: Optional[str] = None,
|
input_reference: Optional[str] = None,
|
||||||
output_path: Optional[str] = None,
|
output_path: Optional[str] = None,
|
||||||
|
extra_fields: Optional[Dict[str, Any]] = None,
|
||||||
) -> Dict[str, Any]:
|
) -> Dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
Generate a video using SGLang Diffusion API and wait for completion.
|
Generate a video using SGLang Diffusion API and wait for completion.
|
||||||
@@ -238,6 +239,10 @@ class SGLDiffusionServerAPI:
|
|||||||
enable_teacache: Enable TEA cache acceleration
|
enable_teacache: Enable TEA cache acceleration
|
||||||
generator_device: Device for random generator ("cuda" or "cpu")
|
generator_device: Device for random generator ("cuda" or "cpu")
|
||||||
input_reference: Path to input reference image for image-to-video
|
input_reference: Path to input reference image for image-to-video
|
||||||
|
extra_fields: Model-specific request fields merged into the payload
|
||||||
|
last, so a caller can reach a model's own request surface
|
||||||
|
(MiniMax-H3's `task`/`conditions`/`target`, per-model flow
|
||||||
|
shifts) without this client growing a parameter per model
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Dictionary containing completed video job information with file_path
|
Dictionary containing completed video job information with file_path
|
||||||
@@ -281,6 +286,10 @@ class SGLDiffusionServerAPI:
|
|||||||
payload["input_reference"] = input_reference
|
payload["input_reference"] = input_reference
|
||||||
if output_path:
|
if output_path:
|
||||||
payload["output_path"] = output_path
|
payload["output_path"] = output_path
|
||||||
|
# merged last so a model-specific field wins over a generic default of
|
||||||
|
# the same name (H3 sizes its output from `target`, not `size`)
|
||||||
|
if extra_fields:
|
||||||
|
payload.update(extra_fields)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Create video generation job
|
# Create video generation job
|
||||||
|
|||||||
@@ -576,6 +576,222 @@ class SGLDiffusionGenerateVideo:
|
|||||||
return (video, video_path)
|
return (video, video_path)
|
||||||
|
|
||||||
|
|
||||||
|
class SGLDiffusionGenerateH3:
|
||||||
|
"""Node to generate joint video and audio with MiniMax-H3.
|
||||||
|
|
||||||
|
H3 denoises a packed video+audio sequence in one pass and routes its
|
||||||
|
conditioning by task rather than by a single reference slot, so it needs
|
||||||
|
its own request shape (`task` / `conditions` / `target`) that the generic
|
||||||
|
video node does not model. The returned MP4 carries both streams.
|
||||||
|
"""
|
||||||
|
|
||||||
|
TASKS = ["t2va", "fl2va", "ref2va"]
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def INPUT_TYPES(cls):
|
||||||
|
return {
|
||||||
|
"required": {
|
||||||
|
"sgld_client": ("SGLD_CLIENT",),
|
||||||
|
"positive_prompt": (
|
||||||
|
"STRING",
|
||||||
|
{
|
||||||
|
"default": "",
|
||||||
|
"tooltip": "Text prompt. Reference material is addressed "
|
||||||
|
"positionally as <Picture 1>, <Video 1>, <Audio 1>",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"task": (
|
||||||
|
cls.TASKS,
|
||||||
|
{
|
||||||
|
"default": "t2va",
|
||||||
|
"tooltip": "t2va: text only. fl2va: first/last keyframes. "
|
||||||
|
"ref2va: image, video, and audio references",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"optional": {
|
||||||
|
"first_frame": (
|
||||||
|
"IMAGE",
|
||||||
|
{"tooltip": "fl2va: becomes the clip's first frame"},
|
||||||
|
),
|
||||||
|
"last_frame": (
|
||||||
|
"IMAGE",
|
||||||
|
{"tooltip": "fl2va: becomes the clip's last frame"},
|
||||||
|
),
|
||||||
|
"reference_image": (
|
||||||
|
"IMAGE",
|
||||||
|
{
|
||||||
|
"tooltip": "ref2va: guides identity and style; not "
|
||||||
|
"preserved as an endpoint frame"
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"reference_video": (
|
||||||
|
"STRING",
|
||||||
|
{"default": "", "tooltip": "ref2va: path or URL to a video"},
|
||||||
|
),
|
||||||
|
"reference_audio": (
|
||||||
|
"STRING",
|
||||||
|
{"default": "", "tooltip": "ref2va: path or URL to audio"},
|
||||||
|
),
|
||||||
|
"negative_prompt": ("STRING", {"default": ""}),
|
||||||
|
"seed": ("INT", {"default": 1101, "min": -1, "max": 2**32 - 1}),
|
||||||
|
"steps": ("INT", {"default": 50, "min": 1, "max": 100}),
|
||||||
|
"short_edge": ("INT", {"default": 768, "min": 256, "max": 1536}),
|
||||||
|
"aspect_ratio": (
|
||||||
|
["16:9", "9:16", "1:1", "auto"],
|
||||||
|
{"default": "16:9"},
|
||||||
|
),
|
||||||
|
"duration_seconds": (
|
||||||
|
"FLOAT",
|
||||||
|
{"default": 5.0, "min": 4.0, "max": 15.0, "step": 0.5},
|
||||||
|
),
|
||||||
|
"flow_shift": ("FLOAT", {"default": 12.0, "min": 0.0, "max": 30.0}),
|
||||||
|
"audio_flow_shift": (
|
||||||
|
"FLOAT",
|
||||||
|
{"default": 3.0, "min": 0.0, "max": 30.0},
|
||||||
|
),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
RETURN_TYPES = ("VIDEO", "STRING")
|
||||||
|
RETURN_NAMES = ("video", "video_path")
|
||||||
|
FUNCTION = "generate"
|
||||||
|
CATEGORY = "SGLDiffusion"
|
||||||
|
OUTPUT_NODE = False
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _material_uri(value: str) -> str:
|
||||||
|
"""Local paths become file:// URIs; remote URLs are passed through."""
|
||||||
|
if value.startswith(("http://", "https://", "file://")):
|
||||||
|
return value
|
||||||
|
return f"file://{os.path.abspath(value)}"
|
||||||
|
|
||||||
|
def generate(
|
||||||
|
self,
|
||||||
|
sgld_client: SGLDiffusionServerAPI,
|
||||||
|
positive_prompt: str,
|
||||||
|
task: str,
|
||||||
|
first_frame: torch.Tensor = None,
|
||||||
|
last_frame: torch.Tensor = None,
|
||||||
|
reference_image: torch.Tensor = None,
|
||||||
|
reference_video: str = "",
|
||||||
|
reference_audio: str = "",
|
||||||
|
negative_prompt: str = "",
|
||||||
|
seed: int = 1101,
|
||||||
|
steps: int = 50,
|
||||||
|
short_edge: int = 768,
|
||||||
|
aspect_ratio: str = "16:9",
|
||||||
|
duration_seconds: float = 5.0,
|
||||||
|
flow_shift: float = 12.0,
|
||||||
|
audio_flow_shift: float = 3.0,
|
||||||
|
):
|
||||||
|
"""Build H3's task-shaped request and submit it through the server API."""
|
||||||
|
if not positive_prompt:
|
||||||
|
raise ValueError("Prompt cannot be empty")
|
||||||
|
|
||||||
|
# 1. keyframes carry a frame_index and are preserved as endpoints;
|
||||||
|
# references are semantic material and keep request order, because
|
||||||
|
# the prompt addresses them positionally per modality
|
||||||
|
conditions = []
|
||||||
|
if first_frame is not None:
|
||||||
|
conditions.append(
|
||||||
|
{
|
||||||
|
"type": "image",
|
||||||
|
"uri": self._material_uri(get_image_path(first_frame)),
|
||||||
|
"role": "keyframe",
|
||||||
|
"frame_index": 0,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if last_frame is not None:
|
||||||
|
conditions.append(
|
||||||
|
{
|
||||||
|
"type": "image",
|
||||||
|
"uri": self._material_uri(get_image_path(last_frame)),
|
||||||
|
"role": "keyframe",
|
||||||
|
"frame_index": -1,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if reference_image is not None:
|
||||||
|
conditions.append(
|
||||||
|
{
|
||||||
|
"type": "image",
|
||||||
|
"uri": self._material_uri(get_image_path(reference_image)),
|
||||||
|
"role": "reference",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if reference_video:
|
||||||
|
conditions.append(
|
||||||
|
{
|
||||||
|
"type": "video",
|
||||||
|
"uri": self._material_uri(reference_video),
|
||||||
|
"role": "reference",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if reference_audio:
|
||||||
|
conditions.append(
|
||||||
|
{
|
||||||
|
"type": "audio",
|
||||||
|
"uri": self._material_uri(reference_audio),
|
||||||
|
"role": "reference",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# 2. reject wiring the server would reject anyway, but name the input
|
||||||
|
# the user has to change
|
||||||
|
if task == "fl2va" and not (first_frame is not None or last_frame is not None):
|
||||||
|
raise ValueError("fl2va requires first_frame, last_frame, or both")
|
||||||
|
if task == "ref2va" and not conditions:
|
||||||
|
raise ValueError(
|
||||||
|
"ref2va requires at least one of reference_image, "
|
||||||
|
"reference_video, or reference_audio"
|
||||||
|
)
|
||||||
|
if task == "t2va" and conditions:
|
||||||
|
raise ValueError("t2va takes no conditioning inputs; pick another task")
|
||||||
|
|
||||||
|
# 3. `target` resolves the aligned canvas and frame count; the `size`
|
||||||
|
# the server API always sends is unused by H3
|
||||||
|
extra_fields = {
|
||||||
|
"task": task,
|
||||||
|
"conditions": conditions,
|
||||||
|
"target": {
|
||||||
|
"short_edge": short_edge,
|
||||||
|
"aspect_ratio": aspect_ratio,
|
||||||
|
"duration_seconds": duration_seconds,
|
||||||
|
},
|
||||||
|
"flow_shift": flow_shift,
|
||||||
|
"audio_flow_shift": audio_flow_shift,
|
||||||
|
}
|
||||||
|
|
||||||
|
request_params = {
|
||||||
|
"prompt": positive_prompt,
|
||||||
|
"seconds": int(duration_seconds),
|
||||||
|
"num_inference_steps": steps,
|
||||||
|
"output_path": folder_paths.get_temp_directory(),
|
||||||
|
"extra_fields": extra_fields,
|
||||||
|
}
|
||||||
|
if negative_prompt:
|
||||||
|
request_params["negative_prompt"] = negative_prompt
|
||||||
|
if seed >= 0:
|
||||||
|
request_params["seed"] = seed
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = sgld_client.generate_video(**request_params)
|
||||||
|
except Exception as e:
|
||||||
|
raise RuntimeError(f"Failed to generate MiniMax-H3 video: {str(e)}")
|
||||||
|
|
||||||
|
video_path = response.get("file_path", "")
|
||||||
|
# H3 aligns the canvas server-side, so the resolved size is only known
|
||||||
|
# from the response; short_edge and aspect_ratio cannot reconstruct it
|
||||||
|
resolved_size = response.get("size", "")
|
||||||
|
if resolved_size:
|
||||||
|
width, height = (int(v) for v in resolved_size.split("x"))
|
||||||
|
else:
|
||||||
|
width = height = short_edge
|
||||||
|
video = convert_video_to_comfy_video(video_path, height, width)
|
||||||
|
|
||||||
|
return (video, video_path)
|
||||||
|
|
||||||
|
|
||||||
class SGLDiffusionServerSetLora:
|
class SGLDiffusionServerSetLora:
|
||||||
"""Node to set LoRA adapter for SGLang Diffusion server."""
|
"""Node to set LoRA adapter for SGLang Diffusion server."""
|
||||||
|
|
||||||
@@ -696,6 +912,7 @@ NODE_CLASS_MAPPINGS = {
|
|||||||
"SGLDiffusionServerModel": SGLDiffusionServerModel,
|
"SGLDiffusionServerModel": SGLDiffusionServerModel,
|
||||||
"SGLDiffusionGenerateImage": SGLDiffusionGenerateImage,
|
"SGLDiffusionGenerateImage": SGLDiffusionGenerateImage,
|
||||||
"SGLDiffusionGenerateVideo": SGLDiffusionGenerateVideo,
|
"SGLDiffusionGenerateVideo": SGLDiffusionGenerateVideo,
|
||||||
|
"SGLDiffusionGenerateH3": SGLDiffusionGenerateH3,
|
||||||
"SGLDiffusionServerSetLora": SGLDiffusionServerSetLora,
|
"SGLDiffusionServerSetLora": SGLDiffusionServerSetLora,
|
||||||
"SGLDiffusionServerUnsetLora": SGLDiffusionServerUnsetLora,
|
"SGLDiffusionServerUnsetLora": SGLDiffusionServerUnsetLora,
|
||||||
"SGLDUNETLoader": SGLDUNETLoader,
|
"SGLDUNETLoader": SGLDUNETLoader,
|
||||||
@@ -707,6 +924,7 @@ NODE_DISPLAY_NAME_MAPPINGS = {
|
|||||||
"SGLDiffusionServerModel": "SGLDiffusion Server Model",
|
"SGLDiffusionServerModel": "SGLDiffusion Server Model",
|
||||||
"SGLDiffusionGenerateImage": "SGLDiffusion Generate Image",
|
"SGLDiffusionGenerateImage": "SGLDiffusion Generate Image",
|
||||||
"SGLDiffusionGenerateVideo": "SGLDiffusion Generate Video",
|
"SGLDiffusionGenerateVideo": "SGLDiffusion Generate Video",
|
||||||
|
"SGLDiffusionGenerateH3": "SGLDiffusion Generate MiniMax-H3",
|
||||||
"SGLDiffusionServerSetLora": "SGLDiffusion Server Set LoRA",
|
"SGLDiffusionServerSetLora": "SGLDiffusion Server Set LoRA",
|
||||||
"SGLDiffusionServerUnsetLora": "SGLDiffusion Server Unset LoRA",
|
"SGLDiffusionServerUnsetLora": "SGLDiffusion Server Unset LoRA",
|
||||||
"SGLDUNETLoader": "SGLDiffusion UNET Loader",
|
"SGLDUNETLoader": "SGLDiffusion UNET Loader",
|
||||||
|
|||||||
@@ -0,0 +1,290 @@
|
|||||||
|
"""Tests for the MiniMax-H3 ComfyUI node's request shape.
|
||||||
|
|
||||||
|
These exercise the real node and the real server-API client together and mock
|
||||||
|
only HTTP, so a change on either side of the payload contract fails here.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import sys
|
||||||
|
import types
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import torch
|
||||||
|
|
||||||
|
PLUGIN_DIR = Path(__file__).resolve().parents[1]
|
||||||
|
PKG = "sgld_comfy_under_test"
|
||||||
|
|
||||||
|
|
||||||
|
def _install_comfy_stubs() -> None:
|
||||||
|
"""Stub the ComfyUI runtime modules the plugin imports at module scope.
|
||||||
|
|
||||||
|
The plugin only ever runs inside ComfyUI, so these packages are absent in
|
||||||
|
a plain checkout; stubbing them keeps the request-shape contract testable
|
||||||
|
without a ComfyUI install or a GPU.
|
||||||
|
"""
|
||||||
|
folder_paths = types.ModuleType("folder_paths")
|
||||||
|
folder_paths.get_temp_directory = lambda: "/tmp"
|
||||||
|
sys.modules.setdefault("folder_paths", folder_paths)
|
||||||
|
|
||||||
|
comfy_api = types.ModuleType("comfy_api")
|
||||||
|
comfy_api_input = types.ModuleType("comfy_api.input")
|
||||||
|
|
||||||
|
class VideoInput:
|
||||||
|
pass
|
||||||
|
|
||||||
|
comfy_api_input.VideoInput = VideoInput
|
||||||
|
comfy_api.input = comfy_api_input
|
||||||
|
sys.modules.setdefault("comfy_api", comfy_api)
|
||||||
|
sys.modules.setdefault("comfy_api.input", comfy_api_input)
|
||||||
|
|
||||||
|
comfy = types.ModuleType("comfy")
|
||||||
|
comfy.model_detection = types.ModuleType("comfy.model_detection")
|
||||||
|
comfy.model_management = types.ModuleType("comfy.model_management")
|
||||||
|
|
||||||
|
comfy_utils = types.ModuleType("comfy.utils")
|
||||||
|
for name in (
|
||||||
|
"calculate_parameters",
|
||||||
|
"load_torch_file",
|
||||||
|
"state_dict_prefix_replace",
|
||||||
|
"unet_to_diffusers",
|
||||||
|
):
|
||||||
|
setattr(comfy_utils, name, lambda *a, **k: None)
|
||||||
|
comfy.utils = comfy_utils
|
||||||
|
|
||||||
|
comfy_model_patcher = types.ModuleType("comfy.model_patcher")
|
||||||
|
|
||||||
|
class ModelPatcher:
|
||||||
|
def __init__(self, *a, **k):
|
||||||
|
pass
|
||||||
|
|
||||||
|
comfy_model_patcher.ModelPatcher = ModelPatcher
|
||||||
|
comfy.model_patcher = comfy_model_patcher
|
||||||
|
|
||||||
|
sys.modules.setdefault("comfy", comfy)
|
||||||
|
sys.modules.setdefault("comfy.model_detection", comfy.model_detection)
|
||||||
|
sys.modules.setdefault("comfy.model_management", comfy.model_management)
|
||||||
|
sys.modules.setdefault("comfy.utils", comfy_utils)
|
||||||
|
sys.modules.setdefault("comfy.model_patcher", comfy_model_patcher)
|
||||||
|
|
||||||
|
|
||||||
|
def _load(module_name: str, relative_path: str):
|
||||||
|
"""Load one plugin source file into the synthetic package."""
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
module_name, PLUGIN_DIR / relative_path
|
||||||
|
)
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
sys.modules[module_name] = module
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def _load_plugin():
|
||||||
|
"""Load the node and its client without importing the sglang package root.
|
||||||
|
|
||||||
|
`sglang/__init__` pulls in the whole LLM serving stack, none of which this
|
||||||
|
contract depends on. `core.generator` is replaced by a placeholder because
|
||||||
|
it reaches for ComfyUI's model machinery and the node never calls it on the
|
||||||
|
server path.
|
||||||
|
"""
|
||||||
|
_install_comfy_stubs()
|
||||||
|
|
||||||
|
package = types.ModuleType(PKG)
|
||||||
|
package.__path__ = [str(PLUGIN_DIR)]
|
||||||
|
sys.modules[PKG] = package
|
||||||
|
|
||||||
|
server_api = _load(f"{PKG}.core.server_api", "core/server_api.py")
|
||||||
|
|
||||||
|
core = types.ModuleType(f"{PKG}.core")
|
||||||
|
core.__path__ = [str(PLUGIN_DIR / "core")]
|
||||||
|
core.SGLDiffusionServerAPI = server_api.SGLDiffusionServerAPI
|
||||||
|
core.SGLDiffusionGenerator = object
|
||||||
|
sys.modules[f"{PKG}.core"] = core
|
||||||
|
|
||||||
|
_load(f"{PKG}.utils", "utils.py")
|
||||||
|
nodes = _load(f"{PKG}.nodes", "nodes.py")
|
||||||
|
return server_api, nodes
|
||||||
|
|
||||||
|
|
||||||
|
SERVER_API, NODES = _load_plugin()
|
||||||
|
SGLDiffusionServerAPI = SERVER_API.SGLDiffusionServerAPI
|
||||||
|
SGLDiffusionGenerateH3 = NODES.SGLDiffusionGenerateH3
|
||||||
|
|
||||||
|
RESOLVED_SIZE = "1344x768"
|
||||||
|
|
||||||
|
|
||||||
|
class _Response:
|
||||||
|
def __init__(self, payload):
|
||||||
|
self._payload = payload
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._payload
|
||||||
|
|
||||||
|
|
||||||
|
def _run_node(**node_kwargs):
|
||||||
|
"""Drive the node through the real client and capture the POST payload."""
|
||||||
|
client = SGLDiffusionServerAPI(base_url="http://127.0.0.1:30010")
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
def fake_post(url, json=None, headers=None, timeout=None):
|
||||||
|
captured.update(json)
|
||||||
|
return _Response({"id": "job-1"})
|
||||||
|
|
||||||
|
def fake_get(url, headers=None, timeout=None):
|
||||||
|
return _Response(
|
||||||
|
{
|
||||||
|
"id": "job-1",
|
||||||
|
"status": "completed",
|
||||||
|
"size": RESOLVED_SIZE,
|
||||||
|
"file_path": "/tmp/out.mp4",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
node = SGLDiffusionGenerateH3()
|
||||||
|
with mock.patch(
|
||||||
|
f"{PKG}.core.server_api.requests.post",
|
||||||
|
side_effect=fake_post,
|
||||||
|
), mock.patch(
|
||||||
|
f"{PKG}.core.server_api.requests.get",
|
||||||
|
side_effect=fake_get,
|
||||||
|
), mock.patch(
|
||||||
|
f"{PKG}.nodes.get_image_path",
|
||||||
|
side_effect=lambda image: "/tmp/frame.png",
|
||||||
|
):
|
||||||
|
result = node.generate(sgld_client=client, **node_kwargs)
|
||||||
|
return captured, result
|
||||||
|
|
||||||
|
|
||||||
|
def _image():
|
||||||
|
return torch.zeros(1, 8, 8, 3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_t2va_sends_task_target_and_flow_shifts():
|
||||||
|
payload, _ = _run_node(positive_prompt="a cat", task="t2va")
|
||||||
|
|
||||||
|
assert payload["task"] == "t2va"
|
||||||
|
assert payload["conditions"] == []
|
||||||
|
assert payload["target"] == {
|
||||||
|
"short_edge": 768,
|
||||||
|
"aspect_ratio": "16:9",
|
||||||
|
"duration_seconds": 5.0,
|
||||||
|
}
|
||||||
|
assert payload["flow_shift"] == 12.0
|
||||||
|
assert payload["audio_flow_shift"] == 3.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_fl2va_maps_keyframes_to_frame_indices():
|
||||||
|
payload, _ = _run_node(
|
||||||
|
positive_prompt="continue the shot",
|
||||||
|
task="fl2va",
|
||||||
|
first_frame=_image(),
|
||||||
|
last_frame=_image(),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert [c["role"] for c in payload["conditions"]] == ["keyframe", "keyframe"]
|
||||||
|
assert [c["frame_index"] for c in payload["conditions"]] == [0, -1]
|
||||||
|
assert all(c["uri"].startswith("file:///") for c in payload["conditions"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_ref2va_preserves_modality_order_for_prompt_tags():
|
||||||
|
payload, _ = _run_node(
|
||||||
|
positive_prompt="use <Picture 1> and <Audio 1>",
|
||||||
|
task="ref2va",
|
||||||
|
reference_image=_image(),
|
||||||
|
reference_video="/data/clip.mp4",
|
||||||
|
reference_audio="/data/voice.mp3",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert [c["type"] for c in payload["conditions"]] == ["image", "video", "audio"]
|
||||||
|
assert {c["role"] for c in payload["conditions"]} == {"reference"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_reference_urls_pass_through_unchanged():
|
||||||
|
url = "https://example.com/clip.mp4"
|
||||||
|
payload, _ = _run_node(
|
||||||
|
positive_prompt="follow <Video 1>",
|
||||||
|
task="ref2va",
|
||||||
|
reference_video=url,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert payload["conditions"][0]["uri"] == url
|
||||||
|
|
||||||
|
|
||||||
|
def test_node_reports_the_server_resolved_canvas():
|
||||||
|
_, (video, video_path) = _run_node(positive_prompt="a cat", task="t2va")
|
||||||
|
|
||||||
|
assert video.get_dimensions() == (1344, 768)
|
||||||
|
assert video_path == "/tmp/out.mp4"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"kwargs,message",
|
||||||
|
[
|
||||||
|
({"task": "fl2va"}, "fl2va requires"),
|
||||||
|
({"task": "ref2va"}, "ref2va requires"),
|
||||||
|
({"task": "t2va", "reference_video": "/data/clip.mp4"}, "t2va takes no"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_task_and_conditioning_must_agree(kwargs, message):
|
||||||
|
with pytest.raises(ValueError, match=message):
|
||||||
|
_run_node(positive_prompt="a cat", **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_extra_fields_win_over_generic_defaults():
|
||||||
|
"""A model's own field must not be shadowed by a same-named generic default."""
|
||||||
|
client = SGLDiffusionServerAPI(base_url="http://127.0.0.1:30010")
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
def fake_post(url, json=None, headers=None, timeout=None):
|
||||||
|
captured.update(json)
|
||||||
|
return _Response({"id": "job-1"})
|
||||||
|
|
||||||
|
with mock.patch(
|
||||||
|
f"{PKG}.core.server_api.requests.post",
|
||||||
|
side_effect=fake_post,
|
||||||
|
), mock.patch(
|
||||||
|
f"{PKG}.core.server_api.requests.get",
|
||||||
|
side_effect=lambda *a, **k: _Response(
|
||||||
|
{"id": "job-1", "status": "completed", "size": RESOLVED_SIZE}
|
||||||
|
),
|
||||||
|
):
|
||||||
|
client.generate_video(
|
||||||
|
prompt="a cat",
|
||||||
|
size="720x1280",
|
||||||
|
extra_fields={"size": "1344x768", "task": "t2va"},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert captured["size"] == "1344x768"
|
||||||
|
assert captured["task"] == "t2va"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("task", ["t2va", "fl2va", "ref2va"])
|
||||||
|
def test_payload_validates_against_the_server_request_model(task):
|
||||||
|
"""The node's payload must satisfy the schema the server actually parses.
|
||||||
|
|
||||||
|
The other tests mock HTTP, so they would still pass if a field were
|
||||||
|
misnamed or mistyped. This one feeds the captured payload to
|
||||||
|
VideoGenerationsRequest, closing that gap without a running server.
|
||||||
|
"""
|
||||||
|
from sglang.multimodal_gen.runtime.entrypoints.openai.protocol import (
|
||||||
|
VideoGenerationsRequest,
|
||||||
|
)
|
||||||
|
|
||||||
|
conditioning = {
|
||||||
|
"t2va": {},
|
||||||
|
"fl2va": {"first_frame": _image()},
|
||||||
|
"ref2va": {"reference_image": _image()},
|
||||||
|
}[task]
|
||||||
|
payload, _ = _run_node(positive_prompt="a cat", task=task, **conditioning)
|
||||||
|
|
||||||
|
request = VideoGenerationsRequest(**payload)
|
||||||
|
|
||||||
|
# the H3 fields ride through as extras; losing them silently would leave a
|
||||||
|
# valid request that generates the wrong thing
|
||||||
|
assert request.task == task
|
||||||
|
assert request.target["short_edge"] == payload["target"]["short_edge"]
|
||||||
|
assert len(request.conditions) == len(payload["conditions"])
|
||||||
Reference in New Issue
Block a user