[diffusion] docs: update ComfyUI sections, trimmed examples, and the RTX 5090 DiT-resident recipe (1.42x) for Qwen-Image-2.1 cookbook (#40573)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Mick Qian
Claude Opus 5
parent
69d1e5cfe0
commit
50ec9702d0
@@ -400,3 +400,9 @@ remains accepted for compatibility, but new clients should use `extra_body`:
|
||||
- `use_resolution_template`: accepted for vLLM-Omni request compatibility.
|
||||
- `use_system_prompt`: whether to add the Cosmos3 system prompt to the chat template.
|
||||
- `guardrails` or `use_guardrails`: per-request guardrail toggle when the server started with guardrails enabled.
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -81,3 +81,9 @@ with open("ernie_image.png", "wb") as f:
|
||||
- `--performance-mode auto` keeps conservative defaults while preserving explicit user flags.
|
||||
- If the checkpoint includes a PE component, SGLang loads it automatically with the native Ministral3 runtime. Use `--layerwise-offload-components pe` when the local PE decoder needs to trade latency for lower GPU memory usage.
|
||||
- Treat FSDP, SP/Ulysses/Ring, and TP as explicit benchmark knobs. Measure the target resolution, step count, and GPU type before making them production defaults.
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="image" />
|
||||
|
||||
@@ -407,3 +407,9 @@ Test Environment:
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="flux" />
|
||||
|
||||
@@ -187,3 +187,9 @@ response = client.images.generate(
|
||||
```
|
||||
|
||||
Base Ideogram 4 presets are `V4_DEFAULT_20`, `V4_QUALITY_48`, and `V4_TURBO_12`. The fal variants automatically select `V4_FAST_20` and `V4_INSTANT_8`, respectively. A preset controls both `num_inference_steps` and guidance, so do not set those fields directly.
|
||||
|
||||
## 5. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="image" />
|
||||
|
||||
@@ -238,3 +238,9 @@ Use `quality=lossless` for this recipe. High-mode output did not pass the separa
|
||||
- For **2-GPU latency**, try **Ulysses SP** (`--num-gpus 2 --ulysses-degree 2`) on both single-shot and multi-shot runs. Use **TP** when you need a different sharding strategy or more than two GPUs.
|
||||
- Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for long multi-shot SP sessions.
|
||||
- JoyEcho outputs per-shot mp4 files with synchronized audio. There is no built-in two-stage HQ upscaling path like LTX-2.3 HQ.
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -334,3 +334,9 @@ Peak Memory Mean (MB): 37466.40
|
||||
Peak Memory Median (MB): 37466.00
|
||||
============================================================
|
||||
```
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="image" />
|
||||
|
||||
@@ -248,3 +248,9 @@ Some community LoRAs only include weights for transformer blocks. In that case,
|
||||
- Use `--ltx2-two-stage-device-mode resident` on high-VRAM GPUs if latency matters more than memory usage.
|
||||
- Use `--ltx2-two-stage-device-mode original` when comparing against official two-stage behavior.
|
||||
- Keep `--width` and `--height` aligned with the target model resolution; for LTX models, these are output video dimensions.
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -184,3 +184,9 @@ LingBot World 2.0 uses raw-frame websocket GT plus per-chunk latency guards for
|
||||
- Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`.
|
||||
- Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks.
|
||||
- Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior.
|
||||
|
||||
## 7. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -159,3 +159,9 @@ LingBot World uses raw-frame websocket GT plus per-chunk latency guards for cons
|
||||
- Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`.
|
||||
- Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks.
|
||||
- Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior.
|
||||
|
||||
## 7. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -117,3 +117,9 @@ The image is used as the first-frame condition.
|
||||
- SGLang supports T2V sizes 1280x704, 704x1280, 832x480, and 480x832.
|
||||
- I2V request images follow the Wan TI2V preprocessing path in SGLang. This is different from the original LongLive dataset resize path.
|
||||
- For multi-shot runs, set `num_frames` to match `len(shot_prompts) * chunks_per_shot * 8` latent frames, that is `num_frames = (len(shot_prompts) * chunks_per_shot * 8 - 1) * 4 + 1`.
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -266,3 +266,9 @@ python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--task image-to-video --dataset vbench --num-prompts 20 --max-concurrency 20 \
|
||||
--port 30002
|
||||
```
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -2055,3 +2055,9 @@ For a measured lower-count AMD deployment, set both `--num-gpus` and
|
||||
`--ulysses-degree` to 4, 2, or 1. AITER packed attention matched segment-wise
|
||||
BF16 SDPA at cosine similarity `0.9999991655` on MI355X and `0.9999991059` on
|
||||
MI300X.
|
||||
|
||||
## 10. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="minimax-h3" />
|
||||
|
||||
@@ -52,6 +52,8 @@ PY
|
||||
|
||||
The picker defaults to native BF16/FP32 precision, exact attention, eager
|
||||
execution, and full-image VAE decoding.
|
||||
Commands omit default values, including one GPU, encoder auto scheduling, and
|
||||
batch size one. Explicit placement and attention overrides preserve each recipe.
|
||||
|
||||
| GPU | Placement / attention | Generation | Edit | Peak VRAM |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -140,29 +142,24 @@ including partly transparent edges, without thresholding or background removal.
|
||||
|
||||
## 4. Offline requests
|
||||
|
||||
Defaults are 1024×1024, 40 steps, CFG 1, and seed 42; output saving is enabled.
|
||||
For GPUs that need offload, also pass the placement flags from the picker.
|
||||
|
||||
### Text-to-image
|
||||
|
||||
```bash Command
|
||||
sglang generate \
|
||||
--model-path /models/qwen-image-2.1 \
|
||||
--model-id Qwen-Image-2.1 \
|
||||
--prompt "A capybara reading a book by candlelight" \
|
||||
--width 1024 --height 1024 \
|
||||
--num-inference-steps 40 --guidance-scale 1 \
|
||||
--seed 0 --save-output
|
||||
--model-path Qwen/Qwen-Image-2.1 \
|
||||
--prompt "A capybara reading a book by candlelight"
|
||||
```
|
||||
|
||||
### Image-conditioned editing
|
||||
|
||||
```bash Command
|
||||
sglang generate \
|
||||
--model-path /models/qwen-image-2.1 \
|
||||
--model-id Qwen-Image-2.1 \
|
||||
--model-path Qwen/Qwen-Image-2.1 \
|
||||
--image-path /path/to/input.png \
|
||||
--prompt "Move the scene to a snowy mountain at sunrise" \
|
||||
--width 1024 --height 1024 \
|
||||
--num-inference-steps 40 --guidance-scale 1 \
|
||||
--seed 0 --save-output
|
||||
--prompt "Move the scene to a snowy mountain at sunrise"
|
||||
```
|
||||
|
||||
Height and width must be positive multiples of 32. Reference images preserve
|
||||
|
||||
@@ -315,3 +315,9 @@ Peak Memory Mean (MB): 47971.49
|
||||
Peak Memory Median (MB): 47971.29
|
||||
============================================================
|
||||
```
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="qwen-image-edit" />
|
||||
|
||||
@@ -428,3 +428,9 @@ Test Environment:
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="qwen-image" />
|
||||
|
||||
@@ -487,3 +487,9 @@ At WebSocket `init` the realtime adapter fills SANA-WM defaults that differ from
|
||||
<Note>
|
||||
`guidance_scale` applies to the dense path (§4) only; the distilled streaming path uses `streaming_cfg_scale` (default `1.0`, i.e. no CFG) so a `guidance_scale` override never accidentally enables CFG on the streaming stage. `denoising_step_list = (1000, 960, 889, 727, 0)` is the official 4-step streaming schedule (it must end in 0).
|
||||
</Note>
|
||||
|
||||
## 10. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -381,3 +381,9 @@ You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 pe
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -459,3 +459,9 @@ Test Environment:
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="video" />
|
||||
|
||||
@@ -366,3 +366,9 @@ Test Environment:
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## 6. Run in ComfyUI
|
||||
|
||||
import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx';
|
||||
|
||||
<ComfyUISupport model="z-image" />
|
||||
|
||||
Reference in New Issue
Block a user