[diffusion] model: support Anima Base v1.0 (#41011)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
This commit is contained in:
@@ -0,0 +1,139 @@
|
||||
---
|
||||
title: Anima
|
||||
description: "Deploy Anima Base v1.0 with SGLang Diffusion for anime and illustration generation, using native components and single- or multi-GPU execution."
|
||||
---
|
||||
|
||||
import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx';
|
||||
import { Deployment } from '/src/snippets/_deployment.jsx';
|
||||
import { config } from '/src/snippets/configs/CircleStone/anima.jsx';
|
||||
|
||||
<DiffusionModelTags tags={["image", "text-to-image", "anime + illustration", "2B transformer"]} />
|
||||
|
||||
## 1. Quick start
|
||||
|
||||
Install the diffusion dependencies on Linux with NVIDIA CUDA, then install this
|
||||
integration from its source checkout:
|
||||
|
||||
```bash Installation
|
||||
uv pip install "sglang[diffusion]" --prerelease=allow
|
||||
uv pip install -e "python[diffusion]"
|
||||
```
|
||||
|
||||
Use the official `circlestone-labs/Anima-Base-v1.0-Diffusers` checkpoint. SGLang reads
|
||||
its `modular_model_index.json` directly; no checkpoint conversion or newer Diffusers
|
||||
runtime is required.
|
||||
|
||||
Start with **Resident + Compiled** for repeated generation on RTX 5090, DGX Spark,
|
||||
or H200. Select **Eager** for shorter startup or frequent shape changes. Spark's
|
||||
128 GB is unified memory shared with the CPU, not dedicated VRAM. See the
|
||||
[measurements and tradeoffs](#5-measured-tuning) below.
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
## 2. Model capabilities
|
||||
|
||||
[Anima](https://huggingface.co/circlestone-labs/Anima) is CircleStone Labs and Comfy
|
||||
Org's text-to-image model for anime and illustration. It combines a Cosmos Predict2
|
||||
transformer with Qwen3 text encoding, a learned T5-token conditioner, and a Qwen-Image
|
||||
VAE. Prompts can combine tags with natural-language descriptions.
|
||||
|
||||
This integration targets **Base v1.0 text-to-image**. It does not claim support for
|
||||
the separate Aesthetic or Turbo checkpoints, single-file ComfyUI weights, or image
|
||||
editing. The checkpoint uses the CircleStone Labs Non-Commercial License; review
|
||||
the [official license](https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md)
|
||||
before deployment.
|
||||
|
||||
## 3. Sampling
|
||||
|
||||
Defaults are 1024 x 1024, 30 steps, guidance scale 4, and an empty negative prompt.
|
||||
Width and height should be divisible by 16. `max_sequence_length` defaults to 512
|
||||
and accepts 1 through 4096. The text conditioner pads short sequences to 512 tokens;
|
||||
these padding positions remain part of the transformer's cross-attention, matching
|
||||
the official implementation.
|
||||
|
||||
For offline generation:
|
||||
|
||||
```bash Generate
|
||||
sglang generate \
|
||||
--model-path circlestone-labs/Anima-Base-v1.0-Diffusers \
|
||||
--prompt "masterpiece, best quality, safe, watercolor landscape, a quiet seaside village at sunset" \
|
||||
--seed 42 \
|
||||
--save-output
|
||||
```
|
||||
|
||||
Use the same seed, generator device, dimensions, scheduler settings, and precision
|
||||
when comparing runtimes. Latents and scheduler updates remain FP32; the transformer,
|
||||
text components, and VAE default to BF16. Different attention kernels can produce
|
||||
small floating-point differences that accumulate during denoising.
|
||||
|
||||
## 4. Runtime features
|
||||
|
||||
The pipeline reuses SGLang's native Qwen3 encoder, Qwen-Image VAE, component loaders,
|
||||
denoising loop, and residency management. Anima's transformer and text conditioner
|
||||
are native modules, not wrappers around Diffusers models.
|
||||
|
||||
For multi-GPU execution, choose TP to shard transformer weights or Ulysses/Ring to
|
||||
shard image tokens. The transformer has 16 heads, so `tp_size * ulysses_degree`
|
||||
must divide 16. CFG parallelism additionally splits conditional and unconditional
|
||||
denoising and requires guidance greater than 1. The total GPU count must match the
|
||||
selected parallel topology.
|
||||
|
||||
CPU and layerwise offload trade memory for transfers. The additional
|
||||
`text_conditioner` component accepts the same residency controls as other native
|
||||
components. Cache-DiT and quantized attention are approximate optimizations; assess
|
||||
image quality for your prompts before enabling them. See
|
||||
[performance optimization](/docs/sglang-diffusion/performance-optimization) for the
|
||||
shared controls. H200 functional checks cover TP, Ulysses, Ring, CFG parallelism,
|
||||
encoder folding, parallel tiled VAE decode, layerwise offload, FlashAttention,
|
||||
Torch SDPA, SageAttention, Cache-DiT, and breakable CUDA graphs. These checks are
|
||||
not a quality guarantee for approximate optimizations. Combined TP and SP,
|
||||
multi-node execution, non-NVIDIA devices, and third-party LoRAs remain unverified.
|
||||
|
||||
Breakable CUDA graphs reuse the conditioner's actual sequence length, without
|
||||
additional text-bucket padding. Requests with uncaptured shapes fall back to eager
|
||||
execution. Standard short prompts use the same 512-token conditioning shape.
|
||||
|
||||
## 5. Measured tuning
|
||||
|
||||
For repeated requests, compilation was faster than eager execution on all three
|
||||
tested platforms. The picker keeps eager available because initial compilation
|
||||
and recompilation for new shapes can take minutes and use additional CPU memory.
|
||||
No CPU offload, quantized attention, or Cache-DiT is enabled in the recommended
|
||||
recipes.
|
||||
|
||||
The following are warm, sequential HTTP request medians: 1024 x 1024, 30 steps,
|
||||
CFG 4, one image, CPU generator, and PNG/base64 output. Startup is excluded.
|
||||
|
||||
| GPU | Recommended recipe | Original eager | Recommended | Peak allocated |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| RTX 5090, 32 GB | cuDNN SDPA + compile, tiled VAE | 6.99 s | 6.10 s | 5.7 GiB |
|
||||
| DGX Spark, 128 GB unified | Torch SDPA + compile, tiled VAE | 26.03 s | 17.41 s | 5.6 GiB |
|
||||
| H200, 141 GB | FlashAttention + compile, untiled VAE | 3.22 s | 2.22 s | 10.2 GiB |
|
||||
|
||||
Measurements used five timed requests after warmup, PyTorch
|
||||
2.13.0+cu130, and the native Anima implementation at `214c9a48bb1` (Spark used
|
||||
`7f3f903f039`, with identical runtime code). Results are workload-specific, not
|
||||
throughput-at-saturation measurements. The original eager baseline uses Torch
|
||||
SDPA on 5090/Spark and FlashAttention on H200, with VAE tiling enabled. Peak
|
||||
allocated memory excludes the CUDA context, reserved pool, and compilation CPU
|
||||
memory; it is not total device usage.
|
||||
|
||||
On **two NVLink-connected H200s**, Auto selects CFG parallelism instead of TP:
|
||||
the compiled, untiled recipe measured **1.28 s**. Untiled decoding trades about
|
||||
4-5 GiB more allocated memory for lower latency and is verified at 1024 x 1024,
|
||||
one output. The tiled recipes additionally passed repeated 512 x 512 and
|
||||
1536 x 1536 requests, plus two outputs at 1024 x 1024. Use **Tiled** for that
|
||||
verified scope. At 512 x 512, two-GPU CFG eager was faster than compiled;
|
||||
more GPUs or compilation are not universally better.
|
||||
|
||||
The recommended recipes change floating-point kernels and, on H200, VAE tiling.
|
||||
They are **not pixel-identical to eager**: across three fixed prompt/seed pairs,
|
||||
the single-GPU recipes above measured PSNR 21.95-35.44 dB and SSIM 0.851-0.969
|
||||
against their same-device eager outputs.
|
||||
These measure output differences, not perceptual quality guarantees. Use eager
|
||||
with the same backend and tiling settings when reproducing an eager reference.
|
||||
|
||||
Breakable CUDA graphs were near parity at 1024 x 1024 while consuming about
|
||||
1.2 GiB more allocated GPU memory for a single captured shape. They are not the
|
||||
default. Cache-DiT can accelerate further, but changes the denoising computation;
|
||||
validate it separately against your quality requirements.
|
||||
@@ -1521,6 +1521,12 @@
|
||||
"cookbook/diffusion/Qwen-Image/Qwen-Image-Edit"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "CircleStone",
|
||||
"pages": [
|
||||
"cookbook/diffusion/CircleStone/Anima"
|
||||
]
|
||||
},
|
||||
{
|
||||
"group": "SenseNova",
|
||||
"tag": "NEW",
|
||||
|
||||
@@ -1865,16 +1865,19 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
&& Number(sel.nodes) === recommendedRecipe.nodes
|
||||
&& Number(sel.gpus_per_node) === recommendedRecipe.gpus_per_node
|
||||
&& sel.topology_mode === "auto"
|
||||
&& ["auto", recommendedRecipe.placement].includes(sel.placement)
|
||||
&& sel.attention === "platform"
|
||||
&& sel.precision === "native"
|
||||
&& ["auto", recommendedRecipe.encoder].includes(sel.encoder)
|
||||
&& sel.execution === "eager";
|
||||
&& serveDims.every((dim) => {
|
||||
const expected = recommendedRecipe[dim.id] ?? dim.default;
|
||||
if (dim.id === "attention") return sel.attention === "platform";
|
||||
return expected === undefined || sel[dim.id] === expected
|
||||
|| (["placement", "encoder"].includes(dim.id) && sel[dim.id] === "auto");
|
||||
});
|
||||
|
||||
const restoreRecommendedRecipe = () => {
|
||||
if (!recommendedRecipe) return;
|
||||
setSel((prev) => reseatHiddenPicks(normalizeBuilderSelection({
|
||||
...prev,
|
||||
...Object.fromEntries(serveDims
|
||||
.map((dim) => [dim.id, recommendedRecipe[dim.id] ?? dim.default ?? dim.options?.[0]?.id])),
|
||||
nodes: recommendedRecipe.nodes,
|
||||
gpus_per_node: recommendedRecipe.gpus_per_node,
|
||||
topology_mode: "auto",
|
||||
@@ -1885,7 +1888,7 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
attention: "platform",
|
||||
precision: "native",
|
||||
encoder: recommendedRecipe.encoder || "auto",
|
||||
execution: "eager",
|
||||
execution: recommendedRecipe.execution || "eager",
|
||||
})));
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
export const config = {
|
||||
modelName: "Anima Base v1.0",
|
||||
supportedHardware: ["rtx5090", "dgx-spark", "h200"],
|
||||
hardware: [
|
||||
{ id: "rtx5090", label: "RTX 5090", vram: "32GB", vendor: "consumer" },
|
||||
{ id: "dgx-spark", label: "DGX Spark", vram: "128GB unified", vendor: "blackwell" },
|
||||
],
|
||||
groupHardware: false,
|
||||
matchDims: [],
|
||||
overlayDims: [
|
||||
{
|
||||
id: "weights", title: "Checkpoint", scope: "base", default: "default",
|
||||
description: "Official self-contained Diffusers checkpoint.",
|
||||
options: [{ id: "default", label: "Base v1.0" }],
|
||||
},
|
||||
{
|
||||
id: "mode", title: "Request mode", scope: "base", default: "text",
|
||||
options: [{ id: "text", label: "Text to image" }],
|
||||
},
|
||||
{
|
||||
id: "placement", title: "Placement", scope: "serve", default: "resident",
|
||||
description: "Keep the pipeline resident when it fits. Use offload for a smaller memory budget.",
|
||||
learnMore: "#4-runtime-features",
|
||||
options: [
|
||||
{ id: "resident", label: "Resident", flags: ["--performance-mode speed"], recommended: true },
|
||||
{ id: "offload", label: "Layerwise offload", flags: ["--performance-mode memory", "--dit-layerwise-offload true"], soft: true, softReason: "This exact HTTP recipe is unverified." },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "execution", title: "Execution", scope: "serve", default: "compile",
|
||||
description: "Compile for repeated requests. Eager avoids compilation at startup and for new shapes.",
|
||||
learnMore: "#5-measured-tuning",
|
||||
options: [
|
||||
{ id: "compile", label: "Compiled", flags: ["--enable-torch-compile true"], recommended: true },
|
||||
{ id: "eager", label: "Eager", flags: [] },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "attention", title: "Attention", scope: "serve", default: "platform",
|
||||
description: "Exact attention backends can differ in floating-point reduction order.",
|
||||
options: [
|
||||
{ id: "platform", label: "Platform default", flags: (s) => [`--attention-backend ${s.hw === "h200" ? "fa" : s.hw === "rtx5090" ? "torch_cudnn_sdpa" : "torch_sdpa"}`], recommended: true },
|
||||
{ id: "fa", label: "FlashAttention", flags: ["--attention-backend fa"], soft: (s) => s.hw !== "h200", softReason: "On RTX 5090 and Spark, this selector falls back to Torch SDPA. Select Torch SDPA explicitly." },
|
||||
{ id: "sdpa", label: "Torch SDPA", flags: ["--attention-backend torch_sdpa"] },
|
||||
{ id: "cudnn", label: "cuDNN SDPA", flags: ["--attention-backend torch_cudnn_sdpa"] },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "cfg", title: "CFG parallelism", scope: "serve", default: "auto",
|
||||
description: "Auto uses CFG parallelism on two H200 GPUs. Manual TP/SP overrides disable automatic CFG splitting.",
|
||||
learnMore: "#5-measured-tuning",
|
||||
options: [
|
||||
{ id: "auto", label: "Auto", flags: [] },
|
||||
{ id: "off", label: "Off", flags: [] },
|
||||
{ id: "on", label: "On", flags: [], disabled: (s) => Number(s.gpus_per_node) % 2 !== 0, disableReason: "CFG parallelism requires an even number of GPUs and guidance greater than 1." },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "vae", title: "VAE decoding", scope: "serve", default: "auto",
|
||||
description: "Auto disables tiling for compiled H200 recipes. Untiled decoding uses about 4-5 GiB more memory at 1024 x 1024.",
|
||||
learnMore: "#5-measured-tuning",
|
||||
options: [
|
||||
{ id: "auto", label: "Auto", flags: [] },
|
||||
{ id: "tiled", label: "Tiled", flags: [] },
|
||||
{ id: "full", label: "Untiled", flags: [] },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "outputs", title: "Outputs", scope: "request", kind: "number",
|
||||
default: 1, min: 1, max: 4, options: [],
|
||||
},
|
||||
],
|
||||
commandBuilder: {
|
||||
defaultSelection: { hw: "rtx5090", nodes: 1, gpus_per_node: 1, topology_mode: "auto", tp_size: 1, ulysses_degree: 1, ring_degree: 1 },
|
||||
resource: {
|
||||
limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 8 } },
|
||||
verifiedRecipes: [
|
||||
{ id: "rtx5090-1gpu-resident-cudnn", hw: "rtx5090", nodes: 1, gpus_per_node: 1, tp_size: 1, ulysses_degree: 1, ring_degree: 1, placement: "resident", attention: "cudnn", execution: "compile", cfg: "auto", vae: "auto" },
|
||||
{ id: "dgx-spark-1gpu-resident-sdpa", hw: "dgx-spark", nodes: 1, gpus_per_node: 1, tp_size: 1, ulysses_degree: 1, ring_degree: 1, placement: "resident", attention: "sdpa", execution: "compile", cfg: "auto", vae: "auto" },
|
||||
{ id: "h200-1gpu-resident-fa", hw: "h200", nodes: 1, gpus_per_node: 1, tp_size: 1, ulysses_degree: 1, ring_degree: 1, placement: "resident", attention: "fa", execution: "compile", cfg: "auto", vae: "auto" },
|
||||
{ id: "h200-2gpu-resident-cfg", hw: "h200", nodes: 1, gpus_per_node: 2, tp_size: 1, ulysses_degree: 1, ring_degree: 1, placement: "resident", attention: "fa", execution: "compile", cfg: "auto", vae: "auto" },
|
||||
],
|
||||
cfgDegree: (s) => s.cfg === "on" || (s.cfg === "auto" && s.topology_mode !== "manual" && s.hw === "h200" && Number(s.gpus_per_node) === 2) ? 2 : 1,
|
||||
vaeTiling: (s) => s.vae === "tiled" || (s.vae === "auto" && (s.hw !== "h200" || s.execution !== "compile")),
|
||||
autoTopology: (s) => ({ tp_size: Number(s.gpus_per_node) / config.commandBuilder.resource.cfgDegree(s), ulysses_degree: 1, ring_degree: 1 }),
|
||||
validateTopology: (s, t) => {
|
||||
const errors = [];
|
||||
if (Number(s.nodes) !== 1) errors.push("This picker covers single-node deployment only.");
|
||||
if (s.hw === "dgx-spark" && Number(s.gpus_per_node) !== 1) errors.push("DGX Spark has one GPU per node.");
|
||||
const cfg = config.commandBuilder.resource.cfgDegree(s);
|
||||
if (!Number.isInteger(t.tp_size) || t.tp_size < 1 || Number(s.nodes) * Number(s.gpus_per_node) !== cfg * t.tp_size * t.ulysses_degree * t.ring_degree) errors.push("GPU count must equal CFG * TP * Ulysses * Ring.");
|
||||
if (16 % (t.tp_size * t.ulysses_degree)) errors.push("TP * Ulysses must divide Anima's 16 attention heads.");
|
||||
if (t.ring_degree > 1 && (s.hw !== "h200" || ["sdpa", "cudnn"].includes(s.attention))) errors.push("Ring requires FlashAttention.");
|
||||
return errors;
|
||||
},
|
||||
},
|
||||
resolveDeployment: (s) => {
|
||||
const r = config.commandBuilder.resource;
|
||||
const attention = s.attention === "platform" ? (s.hw === "h200" ? "fa" : s.hw === "rtx5090" ? "cudnn" : "sdpa") : s.attention;
|
||||
const t = s.topology_mode === "manual"
|
||||
? { tp_size: Number(s.tp_size), ulysses_degree: Number(s.ulysses_degree), ring_degree: Number(s.ring_degree) }
|
||||
: r.autoTopology(s);
|
||||
const errors = r.validateTopology(s, t);
|
||||
const cfg = r.cfgDegree(s);
|
||||
const tiled = r.vaeTiling(s);
|
||||
const world = Number(s.nodes) * Number(s.gpus_per_node);
|
||||
const recipe = r.verifiedRecipes.find((v) => v.hw === s.hw && v.gpus_per_node === Number(s.gpus_per_node) && v.tp_size === t.tp_size && v.ulysses_degree === t.ulysses_degree && v.ring_degree === t.ring_degree && v.placement === s.placement && r.cfgDegree(v) === cfg);
|
||||
const attentions = world > 1 ? ["fa"] : s.execution === "eager"
|
||||
? ["sdpa", "cudnn", ...(s.hw === "h200" ? ["fa"] : [])]
|
||||
: s.hw === "rtx5090" ? ["sdpa", "cudnn"] : [s.hw === "h200" ? "fa" : "sdpa"];
|
||||
const multiOutput = tiled && (s.execution === "compile" || attention === "fa" || (attention === "sdpa" && s.hw !== "h200"));
|
||||
const vaeVerified = tiled || (s.execution === "compile" && attention === (s.hw === "h200" ? "fa" : "sdpa"));
|
||||
const verified = !!recipe && attentions.includes(attention) && vaeVerified && Number(s.outputs) <= (multiOutput ? 2 : 1) && errors.length === 0;
|
||||
const flags = ["--model-path {{MODEL_NAME}}"];
|
||||
if (world > 1) flags.push(`--num-gpus ${world}`, `--tp-size ${t.tp_size}`, `--ulysses-degree ${t.ulysses_degree}`, `--ring-degree ${t.ring_degree}`);
|
||||
if (cfg > 1) flags.push("--enable-cfg-parallel");
|
||||
if (!tiled) flags.push("--vae-tiling false", "--vae-sp false");
|
||||
flags.push("--host {{HOST_IP}}", "--port {{PORT}}");
|
||||
return {
|
||||
match: { hw: s.hw }, nnodes: Number(s.nodes), verified, flags,
|
||||
builder: {
|
||||
topology: t, topologySummary: `CFG ${cfg} / TP ${t.tp_size} / Ulysses ${t.ulysses_degree} / Ring ${t.ring_degree}`,
|
||||
errors, warnings: verified ? [] : ["This exact HTTP recipe has not been verified."],
|
||||
verification: { serve: verified ? "verified" : "unverified", request: verified ? "verified" : "unverified" },
|
||||
resolvedSettings: { attention: s.attention === "platform" ? ({ fa: "FlashAttention", sdpa: "Torch SDPA", cudnn: "cuDNN SDPA" }[attention]) : undefined, cfg: cfg > 1 ? "2-way CFG" : "Off", vae: tiled ? "Tiled" : "Untiled" },
|
||||
},
|
||||
};
|
||||
},
|
||||
},
|
||||
modelNames: { default: "circlestone-labs/Anima-Base-v1.0-Diffusers" },
|
||||
placeholders: {
|
||||
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||
},
|
||||
curl: (s) => `curl -sS http://{{CURL_HOST}}:{{CURL_PORT}}/v1/images/generations \\
|
||||
-H 'Content-Type: application/json' \\
|
||||
-d '${JSON.stringify({ model: "{{MODEL_NAME}}", prompt: "masterpiece, best quality, safe, watercolor landscape, a quiet seaside village at sunset", size: "1024x1024", n: Number(s.outputs), seed: 42, generator_device: "cpu", output_format: "png", response_format: "b64_json" }, null, 2)}'`,
|
||||
cells: [],
|
||||
};
|
||||
@@ -3,6 +3,11 @@
|
||||
export const DiffusionModelCatalog = ({ category }) => {
|
||||
const MODEL_CATALOG = {
|
||||
image: [
|
||||
{
|
||||
name: "Anima",
|
||||
modelIds: ["circlestone-labs/Anima-Base-v1.0-Diffusers"],
|
||||
cookbook: "/cookbook/diffusion/CircleStone/Anima",
|
||||
},
|
||||
{
|
||||
name: "Ming-Image",
|
||||
modelIds: ["inclusionAI/Ming-Image-0.1-Design", "inclusionAI/Ming-Image-0.1-Design-Layer"],
|
||||
|
||||
@@ -43,15 +43,13 @@ def _is_diffusion_model_from_registry(model_path: str) -> bool:
|
||||
|
||||
|
||||
def _is_diffusers_model_dir(model_dir: str) -> bool:
|
||||
"""Check if a local directory contains a valid diffusers model_index.json."""
|
||||
config_path = os.path.join(model_dir, "model_index.json")
|
||||
if not os.path.exists(config_path):
|
||||
return False
|
||||
|
||||
with open(config_path) as f:
|
||||
config = json.load(f)
|
||||
|
||||
return "_diffusers_version" in config
|
||||
"""Check for a standard or modular Diffusers pipeline index."""
|
||||
for filename in ("model_index.json", "modular_model_index.json"):
|
||||
config_path = os.path.join(model_dir, filename)
|
||||
if os.path.isfile(config_path):
|
||||
with open(config_path) as f:
|
||||
return "_diffusers_version" in json.load(f)
|
||||
return False
|
||||
|
||||
|
||||
def _is_gated_diffusion_repo(repo_id: str) -> bool:
|
||||
|
||||
@@ -9,7 +9,7 @@ SGLang diffusion features an end-to-end unified pipeline for accelerating diffus
|
||||
## Key Features
|
||||
|
||||
SGLang Diffusion has the following features:
|
||||
- Broad model support: Wan, FastWan, FLUX, Qwen-Image / Qwen-Image 2.1, LongCat-Image, Z-Image, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3/LTX-2.5, MiniMax-H3, FastH3, VDN-H3, LingBot Video MoE, LingBot World, SANA-Video/SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more
|
||||
- Broad model support: Wan, FastWan, FLUX, Qwen-Image / Qwen-Image 2.1, LongCat-Image, Z-Image, Anima, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3/LTX-2.5, MiniMax-H3, FastH3, VDN-H3, LingBot Video MoE, LingBot World, SANA-Video/SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more
|
||||
- Fast inference speed: empowered by optimized `sgl-kernel` kernels, scheduler/runtime improvements, caching acceleration, and native diffusion hot-path optimizations
|
||||
- Ease of use: OpenAI-compatible api, CLI, and python sdk support
|
||||
- Multi-platform support:
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from sglang.multimodal_gen.configs.models.adapter.base import (
|
||||
AdapterArchConfig,
|
||||
AdapterConfig,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnimaTextConditionerArchConfig(AdapterArchConfig):
|
||||
source_dim: int = 1024
|
||||
target_dim: int = 1024
|
||||
model_dim: int = 1024
|
||||
num_layers: int = 6
|
||||
num_attention_heads: int = 16
|
||||
mlp_ratio: float = 4.0
|
||||
target_vocab_size: int = 32128
|
||||
use_self_attention: bool = True
|
||||
use_layer_norm: bool = False
|
||||
min_sequence_length: int = 512
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnimaTextConditionerConfig(AdapterConfig):
|
||||
arch_config: AnimaTextConditionerArchConfig = field(
|
||||
default_factory=AnimaTextConditionerArchConfig
|
||||
)
|
||||
@@ -0,0 +1,34 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.base import DiTArchConfig, DiTConfig
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnimaArchConfig(DiTArchConfig):
|
||||
in_channels: int = 16
|
||||
out_channels: int = 16
|
||||
num_attention_heads: int = 16
|
||||
attention_head_dim: int = 128
|
||||
num_layers: int = 28
|
||||
mlp_ratio: float = 4.0
|
||||
text_embed_dim: int = 1024
|
||||
adaln_lora_dim: int = 256
|
||||
patch_size: tuple[int, int, int] = (1, 2, 2)
|
||||
max_size: tuple[int, int, int] = (128, 240, 240)
|
||||
rope_scale: tuple[float, float, float] = (1.0, 4.0, 4.0)
|
||||
concat_padding_mask: bool = True
|
||||
extra_pos_embed_type: str | None = None
|
||||
use_crossattn_projection: bool = False
|
||||
img_context_dim_in: int | None = None
|
||||
|
||||
def __post_init__(self):
|
||||
super().__post_init__()
|
||||
self.hidden_size = self.num_attention_heads * self.attention_head_dim
|
||||
self.num_channels_latents = self.in_channels
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnimaDiTConfig(DiTConfig):
|
||||
arch_config: AnimaArchConfig = field(default_factory=AnimaArchConfig)
|
||||
prefix: str = "anima"
|
||||
@@ -0,0 +1,91 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.anima import AnimaDiTConfig
|
||||
from sglang.multimodal_gen.configs.models.encoders.qwen3 import Qwen3TextConfig
|
||||
from sglang.multimodal_gen.configs.models.vaes.qwenimage import QwenImageVAEConfig
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.base import (
|
||||
ImagePipelineConfig,
|
||||
ModelTaskType,
|
||||
)
|
||||
|
||||
|
||||
def anima_text_output(outputs, inputs):
|
||||
return outputs.last_hidden_state * inputs.attention_mask.unsqueeze(-1)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnimaPipelineConfig(ImagePipelineConfig):
|
||||
task_type: ModelTaskType = ModelTaskType.T2I
|
||||
native_only_components: tuple[str, ...] = ("transformer", "text_conditioner")
|
||||
dit_config: AnimaDiTConfig = field(default_factory=AnimaDiTConfig)
|
||||
vae_config: QwenImageVAEConfig = field(default_factory=QwenImageVAEConfig)
|
||||
text_encoder_configs: tuple = field(default_factory=lambda: (Qwen3TextConfig(),))
|
||||
text_encoder_precisions: tuple[str, ...] = ("bf16",)
|
||||
preprocess_text_funcs: tuple[Callable | None, ...] = (None,)
|
||||
postprocess_text_funcs: tuple[Callable, ...] = (anima_text_output,)
|
||||
should_use_guidance: bool = False
|
||||
enable_autocast: bool = False
|
||||
vae_precision: str = "bf16"
|
||||
|
||||
def tokenize_prompt(self, prompt, tokenizer, tok_kwargs):
|
||||
if not 1 <= tok_kwargs.get("max_length", 512) <= 4096:
|
||||
raise ValueError("Anima max_sequence_length must be between 1 and 4096")
|
||||
inputs = tokenizer(prompt, **{**tok_kwargs, "padding": "longest"})
|
||||
if inputs.input_ids.shape[1] == 0:
|
||||
inputs["input_ids"] = inputs.input_ids.new_zeros((len(prompt), 1))
|
||||
inputs["attention_mask"] = inputs.attention_mask.new_zeros((len(prompt), 1))
|
||||
return inputs
|
||||
|
||||
def prepare_sigmas(self, sigmas, num_inference_steps):
|
||||
return self._prepare_sigmas(sigmas, num_inference_steps)
|
||||
|
||||
def get_latent_dtype(self, prompt_dtype):
|
||||
return torch.float32
|
||||
|
||||
def expand_conditioning_to_sample_batch(self, batch):
|
||||
count = batch.num_outputs_per_prompt
|
||||
if count > 1:
|
||||
batch.prompt_embeds = [
|
||||
x.repeat_interleave(count, dim=0) for x in batch.prompt_embeds
|
||||
]
|
||||
if batch.do_classifier_free_guidance:
|
||||
batch.negative_prompt_embeds = [
|
||||
x.repeat_interleave(count, dim=0)
|
||||
for x in batch.negative_prompt_embeds
|
||||
]
|
||||
return batch
|
||||
|
||||
# the DiT shards patch tokens, leaving scheduler latents replicated
|
||||
def shard_latents_for_sp(self, batch, latents):
|
||||
return latents, False
|
||||
|
||||
def get_pos_prompt_embeds(self, batch):
|
||||
return batch.prompt_embeds[0]
|
||||
|
||||
def get_neg_prompt_embeds(self, batch):
|
||||
return batch.negative_prompt_embeds[0]
|
||||
|
||||
def get_decode_scale_and_shift(self, device, dtype, vae):
|
||||
config = self.vae_config.arch_config
|
||||
std = torch.tensor(config.latents_std, device=device, dtype=dtype)
|
||||
mean = torch.tensor(config.latents_mean, device=device, dtype=dtype)
|
||||
return std.reciprocal().view(1, -1, 1, 1, 1), mean.view(1, -1, 1, 1, 1)
|
||||
|
||||
def post_decoding(self, frames, server_args):
|
||||
return frames.squeeze(2)
|
||||
|
||||
|
||||
def register():
|
||||
from sglang.multimodal_gen.configs.sample.anima import AnimaSamplingParams
|
||||
from sglang.multimodal_gen.registry import register_configs
|
||||
|
||||
register_configs(
|
||||
sampling_param_cls=AnimaSamplingParams,
|
||||
pipeline_config_cls=AnimaPipelineConfig,
|
||||
hf_model_paths=["circlestone-labs/Anima-Base-v1.0-Diffusers"],
|
||||
model_detectors=[lambda name: name.lower() == "animamodularpipeline"],
|
||||
)
|
||||
@@ -0,0 +1,14 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import SamplingParams
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnimaSamplingParams(SamplingParams):
|
||||
height: int = 1024
|
||||
width: int = 1024
|
||||
num_frames: int = 1
|
||||
num_inference_steps: int = 30
|
||||
guidance_scale: float = 4.0
|
||||
negative_prompt: str = ""
|
||||
@@ -403,6 +403,10 @@ class CustomBlockAdapterSpec:
|
||||
|
||||
# Custom BlockAdapter metadata for models absent from cache-dit's registry.
|
||||
_CUSTOM_BLOCK_ADAPTER_SPECS: dict[str, CustomBlockAdapterSpec] = {
|
||||
"AnimaTransformer3DModel": CustomBlockAdapterSpec(
|
||||
blocks_attr="transformer_blocks",
|
||||
forward_pattern=ForwardPattern.Pattern_3,
|
||||
),
|
||||
"MingImageTransformer2DModel": CustomBlockAdapterSpec(
|
||||
blocks_attr="layers",
|
||||
forward_pattern=ForwardPattern.Pattern_3,
|
||||
|
||||
@@ -1,3 +1,6 @@
|
||||
from sglang.multimodal_gen.configs.models.adapter.anima import (
|
||||
AnimaTextConditionerConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.models.adapter.ltx_2_connector import (
|
||||
LTX2ConnectorConfig,
|
||||
)
|
||||
@@ -10,9 +13,10 @@ from sglang.multimodal_gen.runtime.loader.component_loaders.component_loader imp
|
||||
|
||||
|
||||
class AdapterLoader(PlainStateDictComponentLoader):
|
||||
component_names = ["connectors", "duration_head"]
|
||||
component_names = ["connectors", "duration_head", "text_conditioner"]
|
||||
|
||||
config_classes = {
|
||||
"text_conditioner": AnimaTextConditionerConfig,
|
||||
"connectors": LTX2ConnectorConfig,
|
||||
"duration_head": LTX2DurationHeadConfig,
|
||||
}
|
||||
|
||||
@@ -939,7 +939,7 @@ class AutoProcessorLoader(ComponentLoader):
|
||||
class TokenizerLoader(ComponentLoader):
|
||||
"""Loader for tokenizers."""
|
||||
|
||||
component_names = ["tokenizer", "text_tokenizer"]
|
||||
component_names = ["tokenizer", "text_tokenizer", "t5_tokenizer"]
|
||||
expected_library = "transformers"
|
||||
|
||||
def load_customized(
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
# Copyright 2026 The HuggingFace Team. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Anima's learned T5-token queries over Qwen3 text states."""
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from torch import nn
|
||||
|
||||
from sglang.multimodal_gen.configs.models.adapter.anima import (
|
||||
AnimaTextConditionerConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.layers.attention import LocalAttention
|
||||
from sglang.multimodal_gen.runtime.managers.memory_managers.layerwise_offload import (
|
||||
LayerwiseOffloadableModuleMixin,
|
||||
)
|
||||
|
||||
|
||||
class AnimaConditionerAttention(nn.Module):
|
||||
def __init__(self, config, context_dim, cross_attention=False):
|
||||
super().__init__()
|
||||
dim = config.model_dim
|
||||
self.heads = config.num_attention_heads
|
||||
self.head_dim = dim // self.heads
|
||||
self.q_proj = nn.Linear(dim, dim, bias=False)
|
||||
self.k_proj = nn.Linear(context_dim, dim, bias=False)
|
||||
self.v_proj = nn.Linear(context_dim, dim, bias=False)
|
||||
self.o_proj = nn.Linear(dim, dim, bias=False)
|
||||
self.q_norm = nn.RMSNorm(self.head_dim, eps=1e-6)
|
||||
self.k_norm = nn.RMSNorm(self.head_dim, eps=1e-6)
|
||||
self.attn = LocalAttention(
|
||||
self.heads, self.head_dim, is_cross_attention=cross_attention
|
||||
)
|
||||
|
||||
def forward(self, x, context, mask, rope, context_rope):
|
||||
q = self.q_norm(self.q_proj(x).unflatten(-1, (self.heads, self.head_dim)))
|
||||
k = self.k_norm(self.k_proj(context).unflatten(-1, (self.heads, self.head_dim)))
|
||||
v = self.v_proj(context).unflatten(-1, (self.heads, self.head_dim))
|
||||
cos, sin = rope
|
||||
q1, q2 = q.chunk(2, -1)
|
||||
q = q * cos + torch.cat((-q2, q1), -1) * sin
|
||||
cos, sin = context_rope
|
||||
k1, k2 = k.chunk(2, -1)
|
||||
k = k * cos + torch.cat((-k2, k1), -1) * sin
|
||||
return self.o_proj(self.attn(q, k, v, attn_mask=mask).flatten(2))
|
||||
|
||||
|
||||
class AnimaConditionerBlock(nn.Module):
|
||||
def __init__(self, config):
|
||||
super().__init__()
|
||||
self.use_self_attention = config.use_self_attention
|
||||
norm = nn.LayerNorm if config.use_layer_norm else nn.RMSNorm
|
||||
eps = 1e-5 if config.use_layer_norm else 1e-6
|
||||
dim = config.model_dim
|
||||
if self.use_self_attention:
|
||||
self.norm_self_attn = norm(dim, eps=eps)
|
||||
self.self_attn = AnimaConditionerAttention(config, dim)
|
||||
self.norm_cross_attn = norm(dim, eps=eps)
|
||||
self.cross_attn = AnimaConditionerAttention(
|
||||
config, config.source_dim, cross_attention=True
|
||||
)
|
||||
self.norm_mlp = norm(dim, eps=eps)
|
||||
self.mlp = nn.Sequential(
|
||||
nn.Linear(dim, int(dim * config.mlp_ratio)),
|
||||
nn.GELU(),
|
||||
nn.Linear(int(dim * config.mlp_ratio), dim),
|
||||
)
|
||||
|
||||
def forward(self, x, source, target_mask, source_mask, rope, source_rope):
|
||||
if self.use_self_attention:
|
||||
normed = self.norm_self_attn(x)
|
||||
x = x + self.self_attn(normed, normed, target_mask, rope, rope)
|
||||
x = x + self.cross_attn(
|
||||
self.norm_cross_attn(x), source, source_mask, rope, source_rope
|
||||
)
|
||||
return x + self.mlp(self.norm_mlp(x))
|
||||
|
||||
|
||||
class AnimaTextConditioner(nn.Module, LayerwiseOffloadableModuleMixin):
|
||||
layerwise_offload_dit_group_enabled = False
|
||||
layer_names = ["blocks"]
|
||||
|
||||
def __init__(self, config: AnimaTextConditionerConfig):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.embed = nn.Embedding(config.target_vocab_size, config.target_dim)
|
||||
self.in_proj = (
|
||||
nn.Linear(config.target_dim, config.model_dim)
|
||||
if config.target_dim != config.model_dim
|
||||
else nn.Identity()
|
||||
)
|
||||
self.blocks = nn.ModuleList(
|
||||
[AnimaConditionerBlock(config) for _ in range(config.num_layers)]
|
||||
)
|
||||
self.out_proj = nn.Linear(config.model_dim, config.target_dim)
|
||||
self.norm = nn.RMSNorm(config.target_dim, eps=1e-6)
|
||||
|
||||
def _rope(self, length, device, dtype):
|
||||
dim = self.config.model_dim // self.config.num_attention_heads
|
||||
inv = 1.0 / (10000.0 ** (torch.arange(0, dim, 2, device=device).float() / dim))
|
||||
freqs = torch.outer(torch.arange(length, device=device).float(), inv)
|
||||
freqs = torch.cat((freqs, freqs), dim=-1)[None, :, None, :]
|
||||
return freqs.cos().to(dtype), freqs.sin().to(dtype)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
source_hidden_states,
|
||||
target_input_ids,
|
||||
target_attention_mask,
|
||||
source_attention_mask,
|
||||
):
|
||||
x = self.in_proj(self.embed(target_input_ids).to(source_hidden_states.dtype))
|
||||
rope = self._rope(x.shape[1], x.device, x.dtype)
|
||||
source_rope = self._rope(source_hidden_states.shape[1], x.device, x.dtype)
|
||||
# additive -inf preserves SDPA's zero output for fully masked rows
|
||||
target_mask = torch.zeros_like(target_attention_mask, dtype=x.dtype)
|
||||
target_mask.masked_fill_(target_attention_mask == 0, float("-inf"))
|
||||
source_mask = torch.zeros_like(source_attention_mask, dtype=x.dtype)
|
||||
source_mask.masked_fill_(source_attention_mask == 0, float("-inf"))
|
||||
for block in self.blocks:
|
||||
x = block(
|
||||
x, source_hidden_states, target_mask, source_mask, rope, source_rope
|
||||
)
|
||||
x = self.norm(self.out_proj(x)) * target_attention_mask.unsqueeze(-1).to(
|
||||
x.dtype
|
||||
)
|
||||
return F.pad(x, (0, 0, 0, max(0, self.config.min_sequence_length - x.shape[1])))
|
||||
|
||||
|
||||
EntryClass = AnimaTextConditioner
|
||||
@@ -0,0 +1,329 @@
|
||||
# Copyright 2025 The NVIDIA Team and The HuggingFace Team. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Native Cosmos Predict2 image transformer used by Anima."""
|
||||
|
||||
import math
|
||||
|
||||
import torch
|
||||
from diffusers.models.embeddings import Timesteps
|
||||
from torch import nn
|
||||
|
||||
from sglang.multimodal_gen.configs.models.dits.anima import AnimaDiTConfig
|
||||
from sglang.multimodal_gen.configs.models.fsdp import is_module_list_entry_in
|
||||
from sglang.multimodal_gen.runtime.distributed import get_tp_world_size
|
||||
from sglang.multimodal_gen.runtime.distributed.sp_shard_utils import (
|
||||
gather_seq,
|
||||
shard_like,
|
||||
shard_seq,
|
||||
tail_attn_meta,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.layers.attention import USPAttention
|
||||
from sglang.multimodal_gen.runtime.layers.layernorm import RMSNorm
|
||||
from sglang.multimodal_gen.runtime.layers.linear import (
|
||||
ColumnParallelLinear,
|
||||
ReplicatedLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.managers.memory_managers.layerwise_offload import (
|
||||
LayerwiseOffloadableModuleMixin,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.models.dits.base import CachableDiT
|
||||
|
||||
|
||||
def _is_anima_block(name, module):
|
||||
return is_module_list_entry_in(name, ("transformer_blocks",))
|
||||
|
||||
|
||||
class AnimaPatchEmbed(nn.Module):
|
||||
def __init__(self, config):
|
||||
super().__init__()
|
||||
self.patch_size = config.patch_size
|
||||
channels = config.in_channels + int(config.concat_padding_mask)
|
||||
self.proj = ReplicatedLinear(
|
||||
channels * math.prod(config.patch_size), config.hidden_size, bias=False
|
||||
)
|
||||
|
||||
def forward(self, x):
|
||||
b, c, t, h, w = x.shape
|
||||
pt, ph, pw = self.patch_size
|
||||
x = x.reshape(b, c, t // pt, pt, h // ph, ph, w // pw, pw)
|
||||
x = x.permute(0, 2, 4, 6, 1, 3, 5, 7).flatten(4).flatten(1, 3)
|
||||
return self.proj(x)[0]
|
||||
|
||||
|
||||
class AnimaRotaryEmbedding(nn.Module):
|
||||
def __init__(self, config):
|
||||
super().__init__()
|
||||
d = config.attention_head_dim
|
||||
self.dims = (d - 4 * (d // 6), 2 * (d // 6), 2 * (d // 6))
|
||||
self.scales = config.rope_scale
|
||||
self.patch_size = config.patch_size
|
||||
|
||||
def forward(self, x):
|
||||
shape = [size // patch for size, patch in zip(x.shape[-3:], self.patch_size)]
|
||||
freqs = []
|
||||
for axis, (dim, scale) in enumerate(zip(self.dims, self.scales)):
|
||||
theta = 10000.0 * scale ** (dim / (dim - 2))
|
||||
inv = 1.0 / (
|
||||
theta ** (torch.arange(0, dim, 2, device=x.device).float() / dim)
|
||||
)
|
||||
angles = torch.outer(
|
||||
torch.arange(shape[axis], device=x.device).float(), inv
|
||||
)
|
||||
view = [1, 1, 1, dim // 2]
|
||||
view[axis] = shape[axis]
|
||||
freqs.append(angles.view(view).expand(*shape, dim // 2))
|
||||
angles = torch.cat(freqs * 2, dim=-1).flatten(0, 2)
|
||||
return angles.cos(), angles.sin()
|
||||
|
||||
|
||||
class AnimaTimeEmbedding(nn.Module):
|
||||
def __init__(self, hidden_size):
|
||||
super().__init__()
|
||||
self.time_proj = Timesteps(
|
||||
hidden_size, flip_sin_to_cos=True, downscale_freq_shift=0
|
||||
)
|
||||
self.t_embedder = nn.Module()
|
||||
self.t_embedder.linear_1 = ReplicatedLinear(
|
||||
hidden_size, hidden_size, bias=False
|
||||
)
|
||||
self.t_embedder.linear_2 = ReplicatedLinear(
|
||||
hidden_size, 3 * hidden_size, bias=False
|
||||
)
|
||||
self.norm = RMSNorm(hidden_size, eps=1e-6)
|
||||
|
||||
def forward(self, timestep, dtype):
|
||||
x = self.time_proj(timestep).to(dtype)
|
||||
emb = self.t_embedder.linear_1(x)[0]
|
||||
emb = self.t_embedder.linear_2(torch.nn.functional.silu(emb))[0]
|
||||
return emb, self.norm(x)
|
||||
|
||||
|
||||
class AnimaAdaLayerNorm(nn.Module):
|
||||
def __init__(self, hidden_size, rank, gated=True):
|
||||
super().__init__()
|
||||
self.gated = gated
|
||||
self.norm = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)
|
||||
self.linear_1 = ReplicatedLinear(hidden_size, rank, bias=False)
|
||||
self.linear_2 = ReplicatedLinear(
|
||||
rank, (3 if gated else 2) * hidden_size, bias=False
|
||||
)
|
||||
|
||||
def forward(self, x, embedded_timestep, temb):
|
||||
modulation = self.linear_1(torch.nn.functional.silu(embedded_timestep))[0]
|
||||
modulation = self.linear_2(modulation)[0]
|
||||
modulation = modulation + temb[..., : modulation.shape[-1]]
|
||||
values = modulation.unsqueeze(1).chunk(3 if self.gated else 2, dim=-1)
|
||||
out = self.norm(x) * (1 + values[1]) + values[0]
|
||||
return (out, values[2]) if self.gated else out
|
||||
|
||||
|
||||
class AnimaAttention(nn.Module):
|
||||
def __init__(self, config, cross_attention, quant_config, prefix):
|
||||
super().__init__()
|
||||
self.heads = config.num_attention_heads // get_tp_world_size()
|
||||
self.head_dim = config.attention_head_dim
|
||||
size = config.hidden_size
|
||||
context_size = config.text_embed_dim if cross_attention else size
|
||||
self.to_q = ColumnParallelLinear(
|
||||
size, size, bias=False, quant_config=quant_config, prefix=f"{prefix}.to_q"
|
||||
)
|
||||
self.to_k = ColumnParallelLinear(
|
||||
context_size,
|
||||
size,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.to_k",
|
||||
)
|
||||
self.to_v = ColumnParallelLinear(
|
||||
context_size,
|
||||
size,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.to_v",
|
||||
)
|
||||
self.to_out = nn.ModuleList(
|
||||
[
|
||||
RowParallelLinear(
|
||||
size,
|
||||
size,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.to_out.0",
|
||||
)
|
||||
]
|
||||
)
|
||||
self.norm_q = RMSNorm(self.head_dim, eps=1e-6)
|
||||
self.norm_k = RMSNorm(self.head_dim, eps=1e-6)
|
||||
self.attn = USPAttention(
|
||||
self.heads,
|
||||
self.head_dim,
|
||||
is_cross_attention=cross_attention,
|
||||
skip_sequence_parallel=cross_attention,
|
||||
prefix=prefix,
|
||||
)
|
||||
|
||||
def forward(self, x, context=None, rope=None, attn_mask_meta=None):
|
||||
context = x if context is None else context
|
||||
q = self.to_q(x)[0].unflatten(-1, (self.heads, self.head_dim))
|
||||
k = self.to_k(context)[0].unflatten(-1, (self.heads, self.head_dim))
|
||||
v = self.to_v(context)[0].unflatten(-1, (self.heads, self.head_dim))
|
||||
q, k = self.norm_q(q), self.norm_k(k)
|
||||
if rope is not None:
|
||||
cos, sin = (r[None, :, None, :] for r in rope)
|
||||
# Cosmos uses split-half RoPE in fp32, then rounds once
|
||||
q1, q2 = q.chunk(2, dim=-1)
|
||||
k1, k2 = k.chunk(2, dim=-1)
|
||||
q = (q.float() * cos + torch.cat((-q2, q1), -1).float() * sin).to(q.dtype)
|
||||
k = (k.float() * cos + torch.cat((-k2, k1), -1).float() * sin).to(k.dtype)
|
||||
out = self.attn(q, k, v, attn_mask_meta=attn_mask_meta).flatten(2)
|
||||
return self.to_out[0](out)[0]
|
||||
|
||||
|
||||
class AnimaFeedForward(nn.Module):
|
||||
def __init__(self, config, quant_config, prefix):
|
||||
super().__init__()
|
||||
size = config.hidden_size
|
||||
inner = int(size * config.mlp_ratio)
|
||||
projection = nn.Module()
|
||||
projection.proj = ColumnParallelLinear(
|
||||
size,
|
||||
inner,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.net.0.proj",
|
||||
)
|
||||
self.net = nn.ModuleList(
|
||||
[
|
||||
projection,
|
||||
nn.Identity(),
|
||||
RowParallelLinear(
|
||||
inner,
|
||||
size,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.net.2",
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
def forward(self, x):
|
||||
x = torch.nn.functional.gelu(self.net[0].proj(x)[0])
|
||||
return self.net[2](x)[0]
|
||||
|
||||
|
||||
class AnimaTransformerBlock(nn.Module):
|
||||
def __init__(self, config, quant_config, prefix):
|
||||
super().__init__()
|
||||
size, rank = config.hidden_size, config.adaln_lora_dim
|
||||
self.norm1 = AnimaAdaLayerNorm(size, rank)
|
||||
self.attn1 = AnimaAttention(config, False, quant_config, f"{prefix}.attn1")
|
||||
self.norm2 = AnimaAdaLayerNorm(size, rank)
|
||||
self.attn2 = AnimaAttention(config, True, quant_config, f"{prefix}.attn2")
|
||||
self.norm3 = AnimaAdaLayerNorm(size, rank)
|
||||
self.ff = AnimaFeedForward(config, quant_config, f"{prefix}.ff")
|
||||
|
||||
def forward(
|
||||
self,
|
||||
hidden_states,
|
||||
encoder_hidden_states,
|
||||
embedded_timestep,
|
||||
temb,
|
||||
rope,
|
||||
attn_mask_meta=None,
|
||||
):
|
||||
x, gate = self.norm1(hidden_states, embedded_timestep, temb)
|
||||
hidden_states = hidden_states + gate * self.attn1(
|
||||
x, rope=rope, attn_mask_meta=attn_mask_meta
|
||||
)
|
||||
x, gate = self.norm2(hidden_states, embedded_timestep, temb)
|
||||
hidden_states = hidden_states + gate * self.attn2(
|
||||
x, context=encoder_hidden_states
|
||||
)
|
||||
x, gate = self.norm3(hidden_states, embedded_timestep, temb)
|
||||
return hidden_states + gate * self.ff(x)
|
||||
|
||||
|
||||
class AnimaTransformer3DModel(CachableDiT, LayerwiseOffloadableModuleMixin):
|
||||
_aliases = ["CosmosTransformer3DModel"]
|
||||
_fsdp_shard_conditions = [_is_anima_block]
|
||||
_compile_conditions = [_is_anima_block]
|
||||
layer_names = ["transformer_blocks"]
|
||||
param_names_mapping = {}
|
||||
reverse_param_names_mapping = {}
|
||||
|
||||
def __init__(self, config: AnimaDiTConfig, hf_config, quant_config=None, **kwargs):
|
||||
super().__init__(config=config, hf_config=hf_config)
|
||||
arch = self.config
|
||||
if (
|
||||
arch.extra_pos_embed_type
|
||||
or arch.use_crossattn_projection
|
||||
or arch.img_context_dim_in
|
||||
):
|
||||
raise ValueError(
|
||||
"Only Anima's Cosmos image transformer architecture is supported"
|
||||
)
|
||||
if arch.num_attention_heads % get_tp_world_size():
|
||||
raise ValueError("Anima attention heads must be divisible by TP size")
|
||||
self.hidden_size = arch.hidden_size
|
||||
self.num_attention_heads = arch.num_attention_heads
|
||||
self.num_channels_latents = arch.in_channels
|
||||
self.patch_embed = AnimaPatchEmbed(arch)
|
||||
self.rope = AnimaRotaryEmbedding(arch)
|
||||
self.time_embed = AnimaTimeEmbedding(arch.hidden_size)
|
||||
self.transformer_blocks = nn.ModuleList(
|
||||
[
|
||||
AnimaTransformerBlock(arch, quant_config, f"transformer_blocks.{i}")
|
||||
for i in range(arch.num_layers)
|
||||
]
|
||||
)
|
||||
self.norm_out = AnimaAdaLayerNorm(
|
||||
arch.hidden_size, arch.adaln_lora_dim, gated=False
|
||||
)
|
||||
self.proj_out = ReplicatedLinear(
|
||||
arch.hidden_size, math.prod(arch.patch_size) * arch.out_channels, bias=False
|
||||
)
|
||||
self.__post_init__()
|
||||
|
||||
def forward(self, hidden_states, encoder_hidden_states, timestep, **kwargs):
|
||||
encoder_hidden_states = encoder_hidden_states.to(hidden_states.dtype)
|
||||
b, _, t, h, w = hidden_states.shape
|
||||
pt, ph, pw = self.config.patch_size
|
||||
if t % pt or h % ph or w % pw:
|
||||
raise ValueError(
|
||||
"Anima latent dimensions must be divisible by the patch size"
|
||||
)
|
||||
rope = self.rope(hidden_states)
|
||||
if self.config.concat_padding_mask:
|
||||
hidden_states = torch.cat(
|
||||
[hidden_states, hidden_states.new_zeros(b, 1, t, h, w)], dim=1
|
||||
)
|
||||
hidden_states = self.patch_embed(hidden_states)
|
||||
hidden_states, shard = shard_seq(hidden_states)
|
||||
rope = tuple(shard_like(r, shard, dim=0) for r in rope)
|
||||
attn_mask_meta = tail_attn_meta(shard, b, hidden_states.device)
|
||||
temb, embedded_timestep = self.time_embed(
|
||||
timestep.to(hidden_states.dtype) / 1000, hidden_states.dtype
|
||||
)
|
||||
for block in self.transformer_blocks:
|
||||
hidden_states = block(
|
||||
hidden_states,
|
||||
encoder_hidden_states,
|
||||
embedded_timestep,
|
||||
temb,
|
||||
rope,
|
||||
attn_mask_meta,
|
||||
)
|
||||
hidden_states = self.norm_out(hidden_states, embedded_timestep, temb)
|
||||
hidden_states = self.proj_out(hidden_states)[0]
|
||||
hidden_states = gather_seq(hidden_states, shard.orig_len)
|
||||
# checkpoint output ordering is (ph, pw, pt, c), unlike the input patches
|
||||
hidden_states = hidden_states.reshape(
|
||||
b, t // pt, h // ph, w // pw, ph, pw, pt, -1
|
||||
)
|
||||
return hidden_states.permute(0, 7, 1, 6, 2, 4, 3, 5).reshape(
|
||||
b, self.config.out_channels, t, h, w
|
||||
)
|
||||
|
||||
|
||||
EntryClass = AnimaTransformer3DModel
|
||||
@@ -215,6 +215,10 @@ class Qwen3Attention(nn.Module):
|
||||
k_item = k[batch_index : batch_index + 1]
|
||||
v_item = v[batch_index : batch_index + 1]
|
||||
|
||||
if valid_len == 0:
|
||||
outputs.append(torch.zeros_like(q_item))
|
||||
continue
|
||||
|
||||
real_output = self.attn(
|
||||
q_item[:, :valid_len],
|
||||
k_item[:, :valid_len],
|
||||
@@ -323,6 +327,8 @@ class Qwen3ForCausalLM(TextEncoder):
|
||||
- FSDP sharding for CPU offload
|
||||
"""
|
||||
|
||||
_aliases = ["Qwen3Model"]
|
||||
|
||||
def __init__(self, config: Qwen3TextConfig) -> None:
|
||||
super().__init__(config)
|
||||
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.composed_pipeline_base import (
|
||||
ComposedPipelineBase,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.lora.pipeline import LoRAPipeline
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.input_validation import (
|
||||
InputValidationStage,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.anima import (
|
||||
AnimaTextConditioningStage,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.text_encoding import (
|
||||
TextEncodingStage,
|
||||
)
|
||||
|
||||
|
||||
class AnimaPipeline(LoRAPipeline, ComposedPipelineBase):
|
||||
pipeline_name = "AnimaModularPipeline"
|
||||
_required_config_modules = [
|
||||
"text_encoder",
|
||||
"tokenizer",
|
||||
"t5_tokenizer",
|
||||
"text_conditioner",
|
||||
"vae",
|
||||
"transformer",
|
||||
"scheduler",
|
||||
]
|
||||
|
||||
def create_pipeline_stages(self, server_args):
|
||||
self.add_stage(InputValidationStage())
|
||||
self.add_stage(
|
||||
TextEncodingStage(
|
||||
text_encoders=[self.get_module("text_encoder")],
|
||||
tokenizers=[self.get_module("tokenizer")],
|
||||
)
|
||||
)
|
||||
self.add_stage(
|
||||
AnimaTextConditioningStage(
|
||||
self.get_module("text_conditioner"), self.get_module("t5_tokenizer")
|
||||
)
|
||||
)
|
||||
self.add_standard_timestep_preparation_stage()
|
||||
self.add_standard_latent_preparation_stage()
|
||||
self.add_standard_denoising_stage()
|
||||
self.add_standard_decoding_stage()
|
||||
|
||||
|
||||
EntryClass = AnimaPipeline
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.runtime.managers.forward_context import set_forward_context
|
||||
from sglang.multimodal_gen.runtime.managers.memory_managers.component_manager import (
|
||||
ComponentUse,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.condition_encoding import (
|
||||
ConditionEncodingStage,
|
||||
)
|
||||
|
||||
|
||||
class AnimaTextConditioningStage(ConditionEncodingStage):
|
||||
def __init__(self, conditioner, tokenizer):
|
||||
super().__init__()
|
||||
self.conditioner = conditioner
|
||||
self.tokenizer = tokenizer
|
||||
|
||||
def component_uses(self, server_args, stage_name=None):
|
||||
return [
|
||||
ComponentUse(self._component_stage_name(stage_name), "text_conditioner")
|
||||
]
|
||||
|
||||
@torch.no_grad()
|
||||
def forward(self, batch, server_args):
|
||||
prompts = batch.prompt if isinstance(batch.prompt, list) else [batch.prompt]
|
||||
max_length = batch.max_sequence_length or 512
|
||||
if not 1 <= max_length <= 4096:
|
||||
raise ValueError("Anima max_sequence_length must be between 1 and 4096")
|
||||
with self.use_declared_component(
|
||||
component_name="text_conditioner", module=self.conditioner
|
||||
) as conditioner:
|
||||
self.conditioner = conditioner
|
||||
batch.prompt_embeds = [
|
||||
self._condition(
|
||||
conditioner,
|
||||
prompts,
|
||||
batch.prompt_embeds[0],
|
||||
batch.prompt_attention_mask[0],
|
||||
max_length,
|
||||
)
|
||||
]
|
||||
if batch.do_classifier_free_guidance:
|
||||
negative = [batch.negative_prompt] * len(prompts)
|
||||
batch.negative_prompt_embeds = [
|
||||
self._condition(
|
||||
conditioner,
|
||||
negative,
|
||||
batch.negative_prompt_embeds[0],
|
||||
batch.negative_attention_mask[0],
|
||||
max_length,
|
||||
)
|
||||
]
|
||||
# Cosmos attends to conditioner padding; additional BCG padding is not valid
|
||||
batch.prompt_attention_mask = None
|
||||
batch.negative_attention_mask = None
|
||||
batch.prompt_embeds_mask = None
|
||||
batch.negative_prompt_embeds_mask = None
|
||||
batch.prompt_seq_lens = [
|
||||
[batch.prompt_embeds[0].shape[1]] * batch.prompt_embeds[0].shape[0]
|
||||
]
|
||||
if batch.do_classifier_free_guidance:
|
||||
batch.negative_prompt_seq_lens = [
|
||||
[batch.negative_prompt_embeds[0].shape[1]]
|
||||
* batch.negative_prompt_embeds[0].shape[0]
|
||||
]
|
||||
return batch
|
||||
|
||||
def _condition(self, conditioner, prompts, embeds, source_mask, max_length):
|
||||
tokens = self.tokenizer(
|
||||
prompts,
|
||||
padding="longest",
|
||||
truncation=True,
|
||||
max_length=max_length,
|
||||
return_tensors="pt",
|
||||
).to(embeds.device)
|
||||
embeds = embeds.to(dtype=next(conditioner.parameters()).dtype)
|
||||
with set_forward_context(current_timestep=0, attn_metadata=None):
|
||||
return conditioner(
|
||||
embeds, tokens.input_ids, tokens.attention_mask, source_mask
|
||||
)
|
||||
@@ -190,7 +190,9 @@ DEFAULT_BCG_TEXT_BUCKETS = (64, 128, 256, 512, 1024)
|
||||
|
||||
BREAKABLE_CUDA_GRAPH_SUPPORTED_MODEL_IDS = frozenset(
|
||||
{
|
||||
"anima-base-v1.0-diffusers",
|
||||
"black-forest-labs/flux.1-dev",
|
||||
"circlestone-labs/anima-base-v1.0-diffusers",
|
||||
"comfy-org/ideogram-4",
|
||||
"efficient-large-model/sana1.5_1.6b_1024px_diffusers",
|
||||
"efficient-large-model/sana-video_2b_480p_diffusers",
|
||||
@@ -237,6 +239,7 @@ BREAKABLE_CUDA_GRAPH_SUPPORTED_MODEL_IDS = frozenset(
|
||||
|
||||
BREAKABLE_CUDA_GRAPH_SUPPORTED_PIPELINE_CONFIGS = frozenset(
|
||||
{
|
||||
"AnimaPipelineConfig",
|
||||
"FluxPipelineConfig",
|
||||
"GlmImagePipelineConfig",
|
||||
"Ideogram4PipelineConfig",
|
||||
@@ -793,7 +796,7 @@ class ServerArgs(DisaggServerArgsMixin):
|
||||
return
|
||||
|
||||
logger.warning(
|
||||
"[Diffusion BCG] disabled for %s: only FLUX.1-dev, Ideogram-4, "
|
||||
"[Diffusion BCG] disabled for %s: only Anima Base v1.0, FLUX.1-dev, Ideogram-4, "
|
||||
"jdopensource/JoyAI-Echo, Lightricks/LTX-2, LongCat-Image, "
|
||||
"MiniMax-H3, Qwen/Qwen-Image, Qwen/Qwen-Image-2512, "
|
||||
"Qwen/Qwen-Image-2.1, SANA1.5, "
|
||||
|
||||
@@ -149,13 +149,12 @@ def _is_weight_bearing_diffusers_component(key: str, value: Any) -> bool:
|
||||
|
||||
|
||||
def _get_declared_weight_component_dirs(model_path: str) -> list[str]:
|
||||
model_index_path = os.path.join(model_path, "model_index.json")
|
||||
model_index_path = _local_model_index_path(model_path)
|
||||
if not os.path.exists(model_index_path):
|
||||
return []
|
||||
|
||||
try:
|
||||
with open(model_index_path) as f:
|
||||
model_index = json.load(f)
|
||||
model_index = _read_model_index(model_index_path)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Failed to read model_index.json at %s: %s", model_index_path, exc
|
||||
@@ -374,13 +373,12 @@ def _ci_validate_diffusers_model(model_path: str) -> tuple[bool, bool]:
|
||||
|
||||
def _verify_diffusers_model_complete(path: str) -> bool:
|
||||
"""Check if a diffusers model directory has all required component subdirectories."""
|
||||
config_path = os.path.join(path, "model_index.json")
|
||||
config_path = _local_model_index_path(path)
|
||||
if not os.path.exists(config_path):
|
||||
return False
|
||||
|
||||
try:
|
||||
with open(config_path) as config_file:
|
||||
model_index = json.load(config_file)
|
||||
model_index = _read_model_index(config_path)
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to read model_index.json at %s: %s", config_path, exc)
|
||||
return False
|
||||
@@ -683,6 +681,29 @@ def maybe_download_lora(
|
||||
return target
|
||||
|
||||
|
||||
def _local_model_index_path(model_path: str) -> str:
|
||||
standard = os.path.join(model_path, "model_index.json")
|
||||
if os.path.isfile(standard):
|
||||
return standard
|
||||
return os.path.join(model_path, "modular_model_index.json")
|
||||
|
||||
|
||||
def _read_model_index(path: str) -> dict[str, Any]:
|
||||
with open(path) as f:
|
||||
config = json.load(f)
|
||||
if os.path.basename(path) == "modular_model_index.json":
|
||||
config.pop("_blocks_class_name", None)
|
||||
for name, spec in config.items():
|
||||
if isinstance(spec, list) and len(spec) == 3 and isinstance(spec[2], dict):
|
||||
# native loaders consume the component directories in this snapshot
|
||||
if spec[2].get("subfolder", name) != name:
|
||||
raise ValueError(
|
||||
f"Modular component {name!r} must use its own subfolder"
|
||||
)
|
||||
config[name] = spec[:2]
|
||||
return config
|
||||
|
||||
|
||||
def verify_model_config_and_directory(model_path: str) -> dict[str, Any]:
|
||||
"""
|
||||
Verify that the model directory contains a valid diffusers configuration.
|
||||
@@ -695,16 +716,16 @@ def verify_model_config_and_directory(model_path: str) -> dict[str, Any]:
|
||||
"""
|
||||
|
||||
# Check for model_index.json which is required for diffusers models
|
||||
config_path = os.path.join(model_path, "model_index.json")
|
||||
config_path = _local_model_index_path(model_path)
|
||||
if not os.path.exists(config_path):
|
||||
raise ValueError(
|
||||
f"Model directory {model_path} does not contain model_index.json. "
|
||||
f"Model directory {model_path} does not contain model_index.json "
|
||||
"or modular_model_index.json. "
|
||||
"Only HuggingFace diffusers format is supported."
|
||||
)
|
||||
|
||||
# Load the config
|
||||
with open(config_path) as f:
|
||||
config = json.load(f)
|
||||
config = _read_model_index(config_path)
|
||||
|
||||
# Verify diffusers version exists
|
||||
if "_diffusers_version" not in config:
|
||||
@@ -745,26 +766,37 @@ def verify_model_config_and_directory(model_path: str) -> dict[str, Any]:
|
||||
return cast(dict[str, Any], config)
|
||||
|
||||
|
||||
def _resolve_remote_repo_model_index_path(model_name_or_path: str) -> str:
|
||||
def _resolve_remote_repo_model_index_path(
|
||||
model_name_or_path: str, filename: str = "model_index.json"
|
||||
) -> str:
|
||||
"""Return a local path to a remote repo's ``model_index.json``"""
|
||||
try:
|
||||
# Cache-aware: no local_dir, so the selected Hub reuses its cache and
|
||||
# revalidates the remote file when online.
|
||||
return hf_hub_download(repo_id=model_name_or_path, filename="model_index.json")
|
||||
return hf_hub_download(repo_id=model_name_or_path, filename=filename)
|
||||
except EntryNotFoundError:
|
||||
# Repo exists but has no model_index.json (single-model repo); let the
|
||||
# caller fall through to the single-model path.
|
||||
if filename == "model_index.json":
|
||||
return _resolve_remote_repo_model_index_path(
|
||||
model_name_or_path, "modular_model_index.json"
|
||||
)
|
||||
raise
|
||||
except Exception as online_err:
|
||||
cached_path = None
|
||||
if not envs.SGLANG_USE_MODELSCOPE.get():
|
||||
from huggingface_hub import try_to_load_from_cache
|
||||
|
||||
cached = try_to_load_from_cache(
|
||||
repo_id=model_name_or_path, filename="model_index.json"
|
||||
filenames = (
|
||||
(filename, "modular_model_index.json")
|
||||
if filename == "model_index.json"
|
||||
else (filename,)
|
||||
)
|
||||
if isinstance(cached, str) and os.path.exists(cached):
|
||||
cached_path = cached
|
||||
for candidate in filenames:
|
||||
cached = try_to_load_from_cache(
|
||||
repo_id=model_name_or_path, filename=candidate
|
||||
)
|
||||
if isinstance(cached, str) and os.path.exists(cached):
|
||||
cached_path = cached
|
||||
break
|
||||
if cached_path is not None:
|
||||
logger.warning(
|
||||
"Could not fetch model_index.json for '%s' from the Hugging Face "
|
||||
@@ -815,8 +847,7 @@ def maybe_download_model_index(model_name_or_path: str) -> dict[str, Any]:
|
||||
model_index_path = _resolve_remote_repo_model_index_path(model_name_or_path)
|
||||
|
||||
# Load the model_index.json
|
||||
with open(model_index_path) as f:
|
||||
config: dict[str, Any] = json.load(f)
|
||||
config = _read_model_index(model_index_path)
|
||||
|
||||
# Verify it has the required fields
|
||||
if "_class_name" not in config:
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Opt-in HTTP validation: SGLANG_ANIMA_TEST_MODEL=<HF id or local checkpoint>."""
|
||||
|
||||
import base64
|
||||
import io
|
||||
import os
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from openai import OpenAI
|
||||
from PIL import Image
|
||||
|
||||
from sglang.multimodal_gen.test.server.test_server_utils import ServerManager
|
||||
from sglang.multimodal_gen.test.test_utils import get_dynamic_server_port
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not os.environ.get("SGLANG_ANIMA_TEST_MODEL"),
|
||||
reason="set SGLANG_ANIMA_TEST_MODEL for full-checkpoint HTTP tests",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def server():
|
||||
manager = ServerManager(
|
||||
model=os.environ["SGLANG_ANIMA_TEST_MODEL"],
|
||||
port=get_dynamic_server_port(),
|
||||
extra_args="--num-gpus 1 --performance-mode speed --attention-backend fa",
|
||||
)
|
||||
context = manager.start()
|
||||
try:
|
||||
yield context
|
||||
finally:
|
||||
context.cleanup()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("size,steps,outputs", [(512, 4, 2), (1024, 30, 1)])
|
||||
def test_repeated_generation(server, size, steps, outputs):
|
||||
with OpenAI(
|
||||
api_key="EMPTY", base_url=f"http://127.0.0.1:{server.port}/v1"
|
||||
) as client:
|
||||
requests = []
|
||||
for _ in range(2):
|
||||
result = client.images.generate(
|
||||
model=server.model,
|
||||
prompt="masterpiece, best quality, safe, watercolor landscape, a quiet seaside village at sunset",
|
||||
size=f"{size}x{size}",
|
||||
n=outputs,
|
||||
response_format="b64_json",
|
||||
extra_body={
|
||||
"seed": 42,
|
||||
"num_inference_steps": steps,
|
||||
"guidance_scale": 4.0,
|
||||
"negative_prompt": "",
|
||||
"generator_device": "cpu",
|
||||
"output_format": "png",
|
||||
},
|
||||
)
|
||||
assert len(result.data) == outputs
|
||||
images = []
|
||||
for item in result.data:
|
||||
with Image.open(io.BytesIO(base64.b64decode(item.b64_json))) as image:
|
||||
assert image.size == (size, size)
|
||||
pixels = np.asarray(image.convert("RGB"))
|
||||
assert pixels.std() > 5, "degenerate image"
|
||||
images.append(pixels)
|
||||
requests.append(images)
|
||||
for first, second in zip(*requests):
|
||||
np.testing.assert_array_equal(first, second)
|
||||
if outputs > 1:
|
||||
assert not np.array_equal(requests[0][0], requests[0][1])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(pytest.main([__file__]))
|
||||
@@ -0,0 +1,228 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
import json
|
||||
import sys
|
||||
from contextlib import nullcontext
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from diffusers.models.transformers.transformer_cosmos import CosmosRotaryPosEmbed
|
||||
from transformers import BatchEncoding
|
||||
|
||||
from sglang.cli.utils import get_is_diffusion_model
|
||||
from sglang.multimodal_gen.configs.models.dits.anima import AnimaArchConfig
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.anima import AnimaPipelineConfig
|
||||
from sglang.multimodal_gen.configs.sample.anima import AnimaSamplingParams
|
||||
from sglang.multimodal_gen.registry import get_model_info
|
||||
from sglang.multimodal_gen.runtime.breakable_cuda_graph.prompt_padding import (
|
||||
pad_masked_prompt_kwargs,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.models.dits.anima import AnimaRotaryEmbedding
|
||||
from sglang.multimodal_gen.runtime.models.encoders.qwen3 import Qwen3Attention
|
||||
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry
|
||||
from sglang.multimodal_gen.runtime.pipelines.anima_pipeline import AnimaPipeline
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.anima import (
|
||||
AnimaTextConditioningStage,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs
|
||||
from sglang.multimodal_gen.runtime.utils import hf_diffusers_utils
|
||||
|
||||
|
||||
def test_registry_recognizes_official_and_local_anima(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"sglang.multimodal_gen.registry.maybe_download_model_index",
|
||||
lambda _: {"_class_name": "AnimaModularPipeline"},
|
||||
)
|
||||
get_model_info.cache_clear()
|
||||
for model in ("circlestone-labs/Anima-Base-v1.0-Diffusers", "/models/custom-anima"):
|
||||
info = get_model_info(model)
|
||||
assert info.pipeline_cls is AnimaPipeline
|
||||
assert info.pipeline_config_cls is AnimaPipelineConfig
|
||||
assert info.sampling_param_cls is AnimaSamplingParams
|
||||
get_model_info.cache_clear()
|
||||
assert (
|
||||
ModelRegistry.resolve_model_cls("Qwen3Model")[0].__name__ == "Qwen3ForCausalLM"
|
||||
)
|
||||
assert (
|
||||
ModelRegistry.resolve_model_cls("AnimaTextConditioner")[0].__name__
|
||||
== "AnimaTextConditioner"
|
||||
)
|
||||
|
||||
|
||||
def test_empty_prompt_keeps_a_masked_qwen_token():
|
||||
def tokenizer(prompts, **kwargs):
|
||||
assert kwargs["padding"] == "longest"
|
||||
return BatchEncoding(
|
||||
{
|
||||
"input_ids": torch.empty(len(prompts), 0, dtype=torch.long),
|
||||
"attention_mask": torch.empty(len(prompts), 0, dtype=torch.long),
|
||||
}
|
||||
)
|
||||
|
||||
inputs = AnimaPipelineConfig().tokenize_prompt([""], tokenizer, {})
|
||||
assert inputs.input_ids.shape == (1, 1)
|
||||
assert not inputs.attention_mask.any()
|
||||
|
||||
|
||||
def test_breakable_cuda_graph_stays_enabled_for_anima():
|
||||
args = SimpleNamespace(
|
||||
model_id="circlestone-labs/Anima-Base-v1.0-Diffusers",
|
||||
model_path="/models/Anima-Base-v1.0-Diffusers",
|
||||
enable_breakable_cuda_graph=True,
|
||||
pipeline_config=AnimaPipelineConfig(),
|
||||
warmup_resolutions=["512x512"],
|
||||
)
|
||||
args._is_breakable_cuda_graph_supported_model = lambda: (
|
||||
ServerArgs._is_breakable_cuda_graph_supported_model(args)
|
||||
)
|
||||
ServerArgs._adjust_breakable_cuda_graph_support(args)
|
||||
assert args.enable_breakable_cuda_graph
|
||||
|
||||
|
||||
@pytest.mark.parametrize("length", [512, 600])
|
||||
def test_conditioning_does_not_allow_extra_bcg_padding(length):
|
||||
embeds = torch.randn(1, length, 8)
|
||||
mask = torch.ones(1, length, dtype=torch.bool)
|
||||
batch = SimpleNamespace(
|
||||
prompt="landscape",
|
||||
negative_prompt="",
|
||||
max_sequence_length=1024,
|
||||
prompt_embeds=[embeds],
|
||||
negative_prompt_embeds=[embeds],
|
||||
prompt_attention_mask=[mask],
|
||||
negative_attention_mask=[mask],
|
||||
prompt_embeds_mask=[mask],
|
||||
negative_prompt_embeds_mask=[mask],
|
||||
do_classifier_free_guidance=True,
|
||||
)
|
||||
stage = SimpleNamespace(
|
||||
conditioner=None,
|
||||
use_declared_component=lambda **kwargs: nullcontext(None),
|
||||
_condition=lambda *args: embeds,
|
||||
)
|
||||
AnimaTextConditioningStage.forward(stage, batch, None)
|
||||
for masks in (batch.prompt_embeds_mask, batch.negative_prompt_embeds_mask):
|
||||
kwargs = {"encoder_hidden_states": embeds, "encoder_hidden_states_mask": masks}
|
||||
assert pad_masked_prompt_kwargs(kwargs, (1024,)) is kwargs
|
||||
assert batch.prompt_seq_lens == batch.negative_prompt_seq_lens == [[length]]
|
||||
|
||||
|
||||
def test_qwen_all_masked_row_never_calls_attention_with_empty_kv():
|
||||
def attention(*args):
|
||||
raise AssertionError("all-masked rows have no valid KV")
|
||||
|
||||
module = SimpleNamespace(attn=attention)
|
||||
q = torch.randn(1, 1, 2, 8)
|
||||
out = Qwen3Attention._masked_causal_attention(module, q, q, q, (0,))
|
||||
torch.testing.assert_close(out, torch.zeros_like(q))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("height,width", [(4, 6), (16, 16), (10, 14)])
|
||||
def test_rope_matches_cosmos(height, width):
|
||||
arch = AnimaArchConfig()
|
||||
x = torch.zeros(1, 16, 1, height, width)
|
||||
actual = AnimaRotaryEmbedding(arch)(x)
|
||||
reference = CosmosRotaryPosEmbed(
|
||||
arch.attention_head_dim,
|
||||
max_size=arch.max_size,
|
||||
patch_size=arch.patch_size,
|
||||
rope_scale=arch.rope_scale,
|
||||
)(x)
|
||||
for a, b in zip(actual, reference):
|
||||
torch.testing.assert_close(a, b, atol=0, rtol=0)
|
||||
|
||||
|
||||
def test_latents_and_conditioning_preserve_sample_batch():
|
||||
config = AnimaPipelineConfig()
|
||||
config.vae_config.post_init()
|
||||
batch = SimpleNamespace(
|
||||
height=512,
|
||||
width=768,
|
||||
num_outputs_per_prompt=2,
|
||||
prompt_embeds=[torch.randn(2, 512, 8)],
|
||||
negative_prompt_embeds=[torch.randn(2, 512, 8)],
|
||||
do_classifier_free_guidance=True,
|
||||
)
|
||||
original = batch.prompt_embeds[0].clone()
|
||||
assert config.prepare_latent_shape(batch, 4, 1) == (4, 16, 1, 64, 96)
|
||||
assert config.get_latent_dtype(torch.bfloat16) == torch.float32
|
||||
config.expand_conditioning_to_sample_batch(batch)
|
||||
torch.testing.assert_close(batch.prompt_embeds[0], original.repeat_interleave(2, 0))
|
||||
assert batch.negative_prompt_embeds[0].shape == (4, 512, 8)
|
||||
latents = torch.randn(4, 16, 1, 64, 96)
|
||||
value, sharded = config.shard_latents_for_sp(batch, latents)
|
||||
assert value is latents and not sharded
|
||||
|
||||
|
||||
def test_modular_index_uses_existing_component_loaders(tmp_path, monkeypatch):
|
||||
index = {
|
||||
"_class_name": "AnimaModularPipeline",
|
||||
"_blocks_class_name": "AnimaAutoBlocks",
|
||||
"_diffusers_version": "0.39.0.dev0",
|
||||
"transformer": [
|
||||
"diffusers",
|
||||
"CosmosTransformer3DModel",
|
||||
{"subfolder": "transformer"},
|
||||
],
|
||||
"t5_tokenizer": ["transformers", "T5Tokenizer", {"subfolder": "t5_tokenizer"}],
|
||||
}
|
||||
path = tmp_path / "modular_model_index.json"
|
||||
path.write_text(json.dumps(index))
|
||||
assert get_is_diffusion_model(str(tmp_path))
|
||||
(tmp_path / "transformer").mkdir()
|
||||
(tmp_path / "transformer" / "diffusion_pytorch_model.safetensors").touch()
|
||||
(tmp_path / "t5_tokenizer").mkdir()
|
||||
config = hf_diffusers_utils.verify_model_config_and_directory(str(tmp_path))
|
||||
assert config["transformer"] == index["transformer"][:2]
|
||||
assert "_blocks_class_name" not in config
|
||||
assert hf_diffusers_utils._verify_diffusers_model_complete(str(tmp_path))
|
||||
calls = []
|
||||
|
||||
def download(repo_id, filename):
|
||||
calls.append(filename)
|
||||
if filename != "modular_model_index.json":
|
||||
raise hf_diffusers_utils.EntryNotFoundError("not found")
|
||||
return str(path)
|
||||
|
||||
monkeypatch.setattr(hf_diffusers_utils, "hf_hub_download", download)
|
||||
remote = hf_diffusers_utils.maybe_download_model_index("test/anima")
|
||||
assert remote["transformer"] == config["transformer"]
|
||||
assert calls[-2:] == ["model_index.json", "modular_model_index.json"]
|
||||
(tmp_path / "model_index.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"_class_name": "Legacy",
|
||||
"_diffusers_version": "0.37.0",
|
||||
"transformer": index["transformer"][:2],
|
||||
}
|
||||
)
|
||||
)
|
||||
assert (
|
||||
hf_diffusers_utils.verify_model_config_and_directory(str(tmp_path))[
|
||||
"_class_name"
|
||||
]
|
||||
== "Legacy"
|
||||
)
|
||||
|
||||
|
||||
def test_modular_index_works_from_offline_cache(tmp_path, monkeypatch):
|
||||
path = tmp_path / "modular_model_index.json"
|
||||
path.write_text("{}")
|
||||
|
||||
def offline(**kwargs):
|
||||
raise hf_diffusers_utils.RequestsConnectionError("offline")
|
||||
|
||||
monkeypatch.setattr(hf_diffusers_utils, "hf_hub_download", offline)
|
||||
monkeypatch.setattr(
|
||||
"huggingface_hub.try_to_load_from_cache",
|
||||
lambda repo_id, filename: (
|
||||
str(path) if filename == "modular_model_index.json" else None
|
||||
),
|
||||
)
|
||||
assert hf_diffusers_utils._resolve_remote_repo_model_index_path(
|
||||
"test/anima"
|
||||
) == str(path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(pytest.main([__file__]))
|
||||
@@ -0,0 +1,122 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Small, checkpoint-free numerical contracts against the Cosmos reference."""
|
||||
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from diffusers.models.transformers.transformer_cosmos import CosmosTransformer3DModel
|
||||
|
||||
from sglang.multimodal_gen.configs.models.adapter.anima import (
|
||||
AnimaTextConditionerArchConfig,
|
||||
AnimaTextConditionerConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.models.dits.anima import AnimaDiTConfig
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.anima import AnimaPipelineConfig
|
||||
from sglang.multimodal_gen.runtime.distributed.parallel_state import (
|
||||
cleanup_dist_env_and_memory,
|
||||
maybe_init_distributed_environment_and_model_parallel,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.managers.forward_context import set_forward_context
|
||||
from sglang.multimodal_gen.runtime.models.adapter.anima import AnimaTextConditioner
|
||||
from sglang.multimodal_gen.runtime.models.dits.anima import AnimaTransformer3DModel
|
||||
from sglang.multimodal_gen.runtime.server_args import ServerArgs, server_args
|
||||
from sglang.multimodal_gen.test.single_test_file.component_accuracy.utils import (
|
||||
ensure_distributed_env_defaults,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module", autouse=True)
|
||||
def cleanup_owned_process_group():
|
||||
initialized = torch.distributed.is_initialized()
|
||||
yield
|
||||
if not initialized:
|
||||
cleanup_dist_env_and_memory()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16])
|
||||
@pytest.mark.parametrize("batch_size", [1, 2])
|
||||
@torch.no_grad()
|
||||
def test_transformer_matches_cosmos(dtype, batch_size, monkeypatch):
|
||||
kwargs = dict(
|
||||
in_channels=4,
|
||||
out_channels=4,
|
||||
num_attention_heads=4,
|
||||
attention_head_dim=32,
|
||||
num_layers=2,
|
||||
mlp_ratio=2,
|
||||
text_embed_dim=32,
|
||||
adaln_lora_dim=16,
|
||||
patch_size=(1, 2, 2),
|
||||
max_size=(128, 240, 240),
|
||||
rope_scale=(1.0, 4.0, 4.0),
|
||||
concat_padding_mask=True,
|
||||
extra_pos_embed_type=None,
|
||||
use_crossattn_projection=False,
|
||||
)
|
||||
config = AnimaDiTConfig()
|
||||
config.update_model_arch(kwargs)
|
||||
args = ServerArgs(
|
||||
model_path="circlestone-labs/Anima-Base-v1.0-Diffusers",
|
||||
num_gpus=1,
|
||||
pipeline_config=AnimaPipelineConfig(dit_config=config),
|
||||
attention_backend="torch_sdpa",
|
||||
)
|
||||
monkeypatch.setattr(server_args, "_global_server_args", args)
|
||||
ensure_distributed_env_defaults()
|
||||
maybe_init_distributed_environment_and_model_parallel(tp_size=1, sp_size=1)
|
||||
torch.manual_seed(42)
|
||||
reference = CosmosTransformer3DModel(**kwargs).cuda().to(dtype).eval()
|
||||
model = AnimaTransformer3DModel(config, kwargs).cuda().to(dtype).eval()
|
||||
model.load_state_dict(reference.state_dict(), strict=True)
|
||||
x = torch.randn(batch_size, 4, 1, 8, 12, device="cuda", dtype=dtype)
|
||||
context = torch.randn(batch_size, 7, 32, device="cuda", dtype=dtype)
|
||||
t = torch.linspace(600, 900, batch_size, device="cuda")
|
||||
padding = torch.zeros(1, 1, 8, 12, device="cuda", dtype=dtype)
|
||||
expected = reference(x, t.to(dtype) / 1000, context, padding_mask=padding).sample
|
||||
with set_forward_context(current_timestep=0, attn_metadata=None):
|
||||
actual = model(x, context, t)
|
||||
tolerance = 2e-5 if dtype == torch.float32 else 2e-2
|
||||
torch.testing.assert_close(actual, expected, atol=tolerance, rtol=tolerance)
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def test_conditioner_masks_source_and_zero_pads_output(monkeypatch):
|
||||
args = ServerArgs(
|
||||
model_path="circlestone-labs/Anima-Base-v1.0-Diffusers",
|
||||
num_gpus=1,
|
||||
pipeline_config=AnimaPipelineConfig(),
|
||||
attention_backend="torch_sdpa",
|
||||
)
|
||||
monkeypatch.setattr(server_args, "_global_server_args", args)
|
||||
config = AnimaTextConditionerConfig(
|
||||
arch_config=AnimaTextConditionerArchConfig(
|
||||
source_dim=32,
|
||||
target_dim=32,
|
||||
model_dim=32,
|
||||
num_attention_heads=2,
|
||||
num_layers=2,
|
||||
target_vocab_size=16,
|
||||
)
|
||||
)
|
||||
model = AnimaTextConditioner(config).cuda().bfloat16().eval()
|
||||
source = torch.randn(2, 4, 32, device="cuda", dtype=torch.bfloat16)
|
||||
ids = torch.tensor([[1, 2, 0], [1, 0, 0]], device="cuda")
|
||||
target_mask = ids != 0
|
||||
source_mask = torch.tensor([[1, 1, 0, 0], [0, 0, 0, 0]], device="cuda")
|
||||
changed = source.clone()
|
||||
changed[source_mask == 0] += 10
|
||||
with set_forward_context(current_timestep=0, attn_metadata=None):
|
||||
actual = model(source, ids, target_mask, source_mask)
|
||||
expected = model(changed, ids, target_mask, source_mask)
|
||||
torch.testing.assert_close(actual, expected, atol=0, rtol=0)
|
||||
assert actual.shape == (2, 512, 32)
|
||||
assert torch.isfinite(actual).all()
|
||||
assert not actual[:, 3:].any()
|
||||
assert not actual[:, :3][~target_mask].any()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(pytest.main([__file__]))
|
||||
@@ -301,14 +301,22 @@ class TestBuildCustomBlockAdapter(unittest.TestCase):
|
||||
def test_builds_adapter_for_registered_class(self):
|
||||
module = _import_module_with_stub()
|
||||
blocks = ["block_0", "block_1"]
|
||||
transformer = _make_transformer("ErnieImageTransformer2DModel", blocks)
|
||||
for class_name, blocks_attr in (
|
||||
("ErnieImageTransformer2DModel", "layers"),
|
||||
("MingImageTransformer2DModel", "layers"),
|
||||
("AnimaTransformer3DModel", "transformer_blocks"),
|
||||
):
|
||||
with self.subTest(class_name=class_name):
|
||||
transformer = type(class_name, (), {blocks_attr: blocks})()
|
||||
adapter = module._build_custom_block_adapter(
|
||||
transformer, has_separate_cfg=True
|
||||
)
|
||||
|
||||
adapter = module._build_custom_block_adapter(transformer, has_separate_cfg=True)
|
||||
|
||||
self.assertIsNotNone(adapter)
|
||||
self.assertEqual(adapter.blocks, blocks)
|
||||
self.assertEqual(adapter.forward_pattern, "Pattern_3")
|
||||
self.assertTrue(adapter.has_separate_cfg)
|
||||
self.assertIsNotNone(adapter)
|
||||
self.assertIs(adapter.transformer, transformer)
|
||||
self.assertIs(adapter.blocks, blocks)
|
||||
self.assertEqual(adapter.forward_pattern, "Pattern_3")
|
||||
self.assertTrue(adapter.has_separate_cfg)
|
||||
|
||||
def test_returns_none_for_unknown_class(self):
|
||||
module = _import_module_with_stub()
|
||||
|
||||
@@ -3,6 +3,22 @@
|
||||
"measurement_repeats": 3,
|
||||
"test_image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png",
|
||||
"cases": [
|
||||
{
|
||||
"id": "anima_base_t2i_1024",
|
||||
"model": "circlestone-labs/Anima-Base-v1.0-Diffusers",
|
||||
"task": "text-to-image",
|
||||
"prompt": "masterpiece, best quality, safe, watercolor landscape, a quiet seaside village at sunset",
|
||||
"width": 1024,
|
||||
"height": 1024,
|
||||
"seed": 42,
|
||||
"num_gpus": 1,
|
||||
"frameworks": {
|
||||
"sglang": {
|
||||
"serve_args": "--warmup-mode server --performance-mode speed",
|
||||
"extra_env": {}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "flux1_dev_t2i_1024",
|
||||
"model": "black-forest-labs/FLUX.1-dev",
|
||||
|
||||
Reference in New Issue
Block a user