diff --git a/docs/cookbook/diffusion/FLUX/FLUX.3.mdx b/docs/cookbook/diffusion/FLUX/FLUX.3.mdx index ad8cdbb06d..f3ca64db83 100644 --- a/docs/cookbook/diffusion/FLUX/FLUX.3.mdx +++ b/docs/cookbook/diffusion/FLUX/FLUX.3.mdx @@ -34,6 +34,13 @@ below. Unverified hardware or resource shapes remain selectable but are not measured recommendations. +The picker defaults to resident weights for its 1 MP, batch-1 workload. Its +recommended execution uses BCG only where a matched E2E comparison showed a +material gain: currently single-GB300 distilled NVFP4 T2I. Other combinations +start eager; this graph recipe covers the 80-token text bucket. Changing +precision never happens automatically. Recommendations are best among the +tested settings, not an exhaustive optimum. + FLUX.3 Image Klein is released under the [FLUX Kommunity License v1.0](https://huggingface.co/black-forest-labs/flux-3-image-klein/blob/main/LICENSE.md): @@ -172,6 +179,21 @@ comparison against BF16. These flags remain separately configurable; see the validation boundaries below before combining quantization with other optimizations. +### Breakable CUDA graphs + +BCG reserves additional device memory and captures specific request shapes. +The picker generates `--warmup-resolutions 1024x1024` and matching +`--warmup-sampling-params` for the selected quality. For editing it also sets +`flux3_mode="i2i"` and a server-local reference path: mount a 1024x1024 image +there before starting the server. This path can differ from the client's upload +path. Longer prompt token buckets, other reference dimensions/counts, batches +or quality settings may run eager instead; check for a serving-signature miss +in the server log. + +Changing the picker workload also changes its graph warmup command. Restart +with that command to capture the new workload. Keep eager for variable-shape +traffic, or explicitly validate every shape you intend to capture. + ### LoRA adapters Use the standard LoRA API with native DiT keys, optionally prefixed by `dit.` or @@ -180,7 +202,9 @@ check, including repeatability and exact restoration after unloading; this does not validate every community adapter. For quantized deployments requiring LoRA, pass `--lora-path` at startup to select the generic quantized layout. Attaching LoRA after PrimsTS weight preparation is rejected before replacing any layers. -Quantized LoRA combinations still need separate end-to-end validation. +FP8R and NVFP4 startup adapters also have synthetic rank-4 generation checks +on GB300, including repeatability and exact restoration after unloading. These +do not establish compatibility with arbitrary quantized community adapters. ## 4. Multi-GPU and memory @@ -196,6 +220,11 @@ uses `--num-gpus 2 --tp-size 1 --ulysses-degree 2`. Base also permits CFG parallelism; distilled checkpoints have no CFG branch to parallelize. More GPUs do not guarantee lower latency, particularly over PCIe. +Ulysses inherits all-to-all streaming through the shared attention layer. +Leave its pipeline-group policy automatic; short sequences need not benefit. +TP-only deployments do not use this path. Encoding-while-decoding overlaps +video-frame decoding with video-file encoding, so it does not apply to this +still-image pipeline. Under sequence parallelism every rank holds the full DiT, so combining it with layerwise offload streams the whole model to each GPU every step. TP also @@ -413,13 +442,42 @@ batching rather than assuming a batch-1 recipe fits on a 32 GB card at batch 2. ## 9. Validation boundaries The release BF16 and FP8R checkpoints have single-GB300/H200 generation checks; -NVFP4 has GB300 checks. Base BF16 also has an RTX 5090 check. Each uses eager +NVFP4 has GB300 and RTX 5090 checks. RTX 5090 also has Base BF16 and both +checkpoints' FP8R and Cache-DiT checks. Single-GPU measurements use eager execution, all components resident, batch 1, 1024x1024 T2I and I2I, two prompts/seeds and repeated requests. Base uses 50 steps and distilled uses 16. Repeated requests matched pixel-for-pixel within each tested configuration. This does not establish official-reference parity or validate every combination of hardware, parallelism, batching and optional features in the picker. +### Multi-GPU measurements + +Base BF16, 50 steps, CFG 3.5, batch 1 and `quality="exact"`. These are warmed +localhost HTTP medians through PNG decoding, excluding loading and warmup; +two prompts/seeds and two timed rounds per prompt. Software: PyTorch +2.14.1+cu130, Transformers 5.19.0, FlashInfer 0.7.1. Both runs use checkpoint +`d8acde427a698bdb98b5c1c941c60fa4c97e9f28`. The H200 run uses FA3 and +NVLink; the RTX 4090 run uses Torch SDPA over PCIe without CUDA peer access. + +| GPUs | Topology | T2I 1024x1024 | I2I, 1 MP reference/output | +| --- | --- | --- | --- | +| 2x H200 | TP2 | 4.27 s | Not measured | +| 2x H200 | Ulysses 2, streaming auto | 4.68 s | Not measured | +| 4x RTX 4090 | TP4 | 16.97 s | 32.61 s | +| 4x RTX 4090 | TP2 x Ulysses 2 | 16.28 s | 32.63 s | + +The picker keeps TP as its multi-GPU starting point, not a guarantee that TP +always wins. A hybrid topology can help on PCIe; validate it on your own +interconnect. H100, B200 and other unmeasured resource shapes are extrapolated +starting points and are not labelled verified. + +On the same two H200s, Ulysses streaming reduced 2048x2048 T2I from 24.00 to +22.87 s and editing a 2048x2048 reference into a 1024x1024 output from 31.01 +to 29.54 s. Streamed and non-streamed outputs matched pixel-for-pixel in these +samples, with no quantization or caching. At 1024x1024, automatic dispatch +selected no streaming groups and gave no speedup. TP and SP themselves are +not bit-identical: sample SSIM was 0.9890-0.9985. None of these comparisons +establishes which path is closer to an official reference. ### Single-GPU measurements Warmed local HTTP medians, including PNG encoding and response transfer, from @@ -441,8 +499,50 @@ reference, and reports the lowest value across two samples, not a quality bound. For the distilled checkpoint, quantization's T2I benefit was small on GB300; do not assume Base's speedup transfers to a 16-step workload. Cache-DiT at threshold 0.12 reduced Base I2I from 6.31 to 3.08 s on GB300, with sample SSIM -0.9786-0.9975. It remains opt-in and approximate. Breakable CUDA graphs did not -provide a clear latency win in the tested GB300 workload; eager stays recommended. +0.9786-0.9975. It remains opt-in and approximate. + +### Eager versus CUDA graphs + +Matched ABBA comparisons use `quality="lossless"`, resident weights, batch 1, +1024x1024 output/reference, and the same software versions above with stable +FlashInfer 0.7.1. Base uses 50 steps/CFG 3.5; distilled uses 16 steps/no CFG. +Times are warmed localhost HTTP medians through JPEG decoding: two prompts/seeds +and four timed samples per arm, excluding loading and graph capture. These +JPEG timings are not directly comparable with the earlier PNG table. + +| GPU | Checkpoint | Workload | Eager | BCG | Recommended | +| --- | --- | --- | --- | --- | --- | +| GB300 | Distilled NVFP4, FA4 | T2I | 0.563 s | 0.436 s | BCG | +| GB300 | Distilled NVFP4, FA4 | I2I | 0.882 s | 0.886 s | Eager | +| GB300 | Base BF16, FA4 | T2I | 3.103 s | 3.130 s | Eager | +| GB300 | Base BF16, FA4 | I2I | 6.236 s | 6.733 s | Eager | +| H200 | Base BF16, FA3 | T2I | 6.845 s | 6.916 s | Eager | +| H200 | Base BF16, FA3 | I2I | 14.183 s | 15.252 s | Eager | +| 2x H200, TP2 | Base BF16, FA3 | T2I | 4.000 s | 4.199 s | Eager | +| RTX 5090 | Distilled NVFP4, SDPA | T2I | 1.676 s | 1.678 s | Eager | +| RTX 5090 | Distilled NVFP4, SDPA | I2I | 4.092 s | 4.104 s | Eager | +| RTX 5090 | Base NVFP4, SDPA | T2I | 9.449 s | 9.583 s | Eager | + +GB300 distilled NVFP4 T2I improves by 22.5% with FA4 + BCG. Peak allocated +memory increases from 16.63 to 17.46 GiB; peak reserved rises from 18.75 to +19.96 GiB. These are per-process allocator maxima across warmup/runtime, +not loading or total-device peaks. Other workloads also need headroom; memory +capacity alone is not a reason to enable graphs when they do not improve latency. + +All 15 fixed-backend eager/BCG workload pairs passed repeated-request checks +and pixel-identical PNG comparisons across the two prompts. This is not an +official-reference accuracy guarantee or evidence that quantization preserves +BF16 output. GB300 measurements use Base revision `17333191c36bae1b61b0a247e33ba284f1df07fa` +and distilled `875076bf83d4364b68dccdbf8307f4f01475f68d`; H200/5090 use +`d8acde427a698bdb98b5c1c941c60fa4c97e9f28` and +`4d02a6a4dc4d5c7a0f3fa767187483b2b70f2384`. TP2 uses tensor-identical Base +DiT weights with a different safetensors header. + +GB300 distilled NVFP4 T2I also ran at 0.424 s with SDPA + BCG (versus 0.589 s +SDPA eager). It is selectable but not the default backend: SDPA versus FA4 +outputs had sample SSIM 0.978-0.987 and MAD 1.74-1.75/255. Neither path is +established as closer to the official output. Keep the backend fixed when +evaluating the graph speedup. The native DiT's packed CFG forward has focused comparison tests against the shared FLUX 3 transformer. Such component tests do not replace full-checkpoint diff --git a/docs/src/snippets/configs/black-forest-labs/flux3.jsx b/docs/src/snippets/configs/black-forest-labs/flux3.jsx index c39a792f1a..fbfcaab5b0 100644 --- a/docs/src/snippets/configs/black-forest-labs/flux3.jsx +++ b/docs/src/snippets/configs/black-forest-labs/flux3.jsx @@ -9,6 +9,49 @@ const hardware = [ { id: "rtx6000pro", label: "RTX PRO 6000 Blackwell", vram: "96GB", vendor: "consumer" }, ]; const hardwareLabels = { rtx5090: "RTX 5090", rtx6000pro: "RTX PRO 6000", h100: "H100", h200: "H200", b200: "B200", b300: "B300", gb200: "GB200", gb300: "GB300" }; +// Matched eager/BCG HTTP runs: 1 MP, batch 1, 80-token bucket, resident. +const graphRuns = [ + { hw: "gb300", variant: "distilled", quant: "bf16", mode: "t2i", execution: "eager" }, + { hw: "gb300", variant: "distilled", quant: "nvfp4", mode: "t2i", execution: "bcg" }, + { hw: "gb300", variant: "distilled", quant: "nvfp4", mode: "t2i", execution: "bcg", attention: "sdpa" }, + { hw: "gb300", variant: "base", quant: "bf16", mode: "t2i", execution: "eager" }, + { hw: "gb300", variant: "base", quant: "bf16", mode: "t2i", execution: "eager", attention: "sdpa" }, + { hw: "gb300", variant: "base", quant: "bf16", mode: "i2i", execution: "eager" }, + { hw: "gb300", variant: "distilled", quant: "nvfp4", mode: "i2i", execution: "eager" }, + { hw: "gb300", variant: "base", quant: "nvfp4", mode: "t2i", execution: "eager" }, + { hw: "h200", variant: "distilled", quant: "bf16", mode: "t2i", execution: "eager" }, + { hw: "h200", variant: "base", quant: "bf16", mode: "t2i", execution: "eager" }, + { hw: "h200", variant: "base", quant: "bf16", mode: "i2i", execution: "eager" }, + { hw: "h200", variant: "base", quant: "bf16", mode: "t2i", execution: "eager", gpus_per_node: 2, tp_size: 2 }, + { hw: "rtx5090", variant: "distilled", quant: "nvfp4", mode: "t2i", execution: "eager", attention: "sdpa" }, + { hw: "rtx5090", variant: "distilled", quant: "nvfp4", mode: "i2i", execution: "eager", attention: "sdpa" }, + { hw: "rtx5090", variant: "base", quant: "nvfp4", mode: "t2i", execution: "eager", attention: "sdpa" }, +].map((recipe) => ({ + nodes: 1, gpus_per_node: 1, tp_size: 1, ulysses_degree: 1, ring_degree: 1, + placement: "resident", attention: "fa", quality: "lossless", + batch_size: 1, outputs: 1, ...recipe, +})); +const matches = (recipe, selection) => Object.entries(recipe).every(([key, value]) => + ["id", "default"].includes(key) || String(selection[key]) === String(value)); +const measuredProfiles = [ + ...["gb300", "h200"].flatMap((hw) => ["base", "distilled"].flatMap((variant) => + (hw === "gb300" ? ["bf16", "fp8r", "nvfp4"] : ["bf16", "fp8r"]) + .map((quant) => ({ hw, variant, quant, attention: "fa" })))), + ...["base", "distilled"].flatMap((variant) => + (variant === "base" ? ["bf16", "fp8r", "nvfp4"] : ["fp8r", "nvfp4"]) + .map((quant) => ({ hw: "rtx5090", variant, quant, attention: "sdpa" }))), + ...[{ tp_size: 2, ulysses_degree: 1 }, { tp_size: 1, ulysses_degree: 2 }] + .map((topology) => ({ hw: "h200", variant: "base", quant: "bf16", + attention: "fa", mode: "t2i", gpus_per_node: 2, ...topology })), + ...graphRuns.flatMap((recipe) => ["eager", "bcg"].map((execution) => ({ ...recipe, execution }))), + { hw: "rtx5090", variant: "base", quant: "bf16", mode: "t2i", + attention: "sdpa", quality: "lossless" }, +].map((recipe, index) => ({ + id: `measured-${index}`, nodes: 1, gpus_per_node: 1, + tp_size: 1, ulysses_degree: 1, ring_degree: 1, + placement: "resident", execution: "eager", quality: "exact", + batch_size: 1, outputs: 1, default: index === 0, ...recipe, +})); return { modelName: "FLUX.3", @@ -17,8 +60,8 @@ return { groupHardware: true, matchDims: [], overlayDims: [ - // Request-scope rows only shape the cURL payload: `sglang serve` rejects - // sampling flags, and one server serves both modes. + // Sampling fields belong to cURL; graph recipes also mirror their shapes + // into the server's warmup configuration. { id: "mode", title: "Request mode", @@ -80,9 +123,9 @@ return { scope: "serve", docsHref: "/docs/sglang-diffusion/api/cli#component-residency", description: "A 1 MP, batch-1 workload fits resident on RTX 5090. Larger references or batches need additional headroom; use auto or offload when necessary.", - default: "auto", + default: "resident", options: [ - { id: "auto", label: "Auto", recommended: true, flags: [] }, + { id: "auto", label: "Auto", flags: [] }, { id: "offload", label: "CPU offload", @@ -101,19 +144,20 @@ return { }, { id: "attention", title: "Attention", scope: "serve", default: "auto", - description: "Keep the backend fixed for numerical comparisons.", + description: "Recommended uses the measured recipe's backend; otherwise FlashAttention on datacenter GPUs and Torch SDPA on RTX 5090. Unmeasured settings remain unverified.", options: [ - { id: "auto", label: "Platform default", flags: [], recommended: true }, - { id: "sdpa", label: "Torch SDPA", flags: ["--attention-backend torch_sdpa"] }, - { id: "fa", label: "FlashAttention", flags: ["--attention-backend fa"] }, + { id: "auto", label: "Recommended", flags: [], recommended: true }, + { id: "sdpa", label: "Torch SDPA", flags: [] }, + { id: "fa", label: "FlashAttention", flags: [] }, ], }, { - id: "execution", title: "Execution", scope: "serve", default: "eager", - description: "CUDA graphs can reduce launch overhead but reserve additional memory.", + id: "execution", title: "Execution", scope: "serve", default: "auto", + description: "Recommended uses BCG only for a measured latency win with resident weights and batch 1. Graphs reserve extra memory and replay only captured input shapes.", options: [ - { id: "eager", label: "Eager", flags: [], recommended: true }, - { id: "bcg", label: "Breakable CUDA graphs", flags: ["--enable-breakable-cuda-graph"], soft: true, softReason: "Not verified for every hardware and topology." }, + { id: "auto", label: "Recommended", flags: [], recommended: true }, + { id: "eager", label: "Eager", flags: [] }, + { id: "bcg", label: "Breakable CUDA graphs", flags: [] }, ], }, { @@ -181,13 +225,8 @@ return { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 8 }, }, - verifiedRecipes: [{ - id: "gb300-base-bf16", hw: "gb300", nodes: 1, - gpus_per_node: 1, tp_size: 1, ulysses_degree: 1, ring_degree: 1, - variant: "base", quant: "bf16", placement: "resident", attention: "fa", - execution: "eager", quality: "exact", batch_size: 1, outputs: 1, - default: true, - }], + verifiedRecipes: measuredProfiles.filter((recipe) => + recipe.variant === "base" && recipe.quant === "bf16" && recipe.ulysses_degree === 1 && recipe.quality === "exact"), autoTopology: (s) => ({ tp_size: Number(s.gpus_per_node) || 1, ulysses_degree: 1, @@ -233,17 +272,29 @@ return { if (Number(s.batch_size) > 1) { parallel.push(`--batching-max-size ${Number(s.batch_size)}`, `--batching-delay-ms ${Number(s.batch_delay) || 0}`); } - const testedProfile = (["gb300", "h200"].includes(s.hw) && s.attention === "fa" - && ["base", "distilled"].includes(s.variant) - && (s.quant === "bf16" || s.quality === "exact") - && (s.hw === "gb300" || s.quant !== "nvfp4")) - || (s.hw === "rtx5090" && s.attention === "sdpa" - && s.variant === "base" && s.quant === "bf16"); - const verified = testedProfile && gpus === 1 - && s.placement === "resident" - && s.execution === "eager" && ["exact", "lossless"].includes(s.quality) - && Number(s.batch_size) === 1 && Number(s.outputs) === 1; + const preferred = graphRuns.find(({ attention, execution, ...shape }) => matches(shape, { ...s, ...topology })); + const attention = s.attention === "auto" + ? preferred?.attention || (s.hw === "rtx5090" ? "sdpa" : "fa") : s.attention; + const selected = { ...s, ...topology, attention }; + const best = graphRuns.find(({ execution, ...shape }) => matches(shape, selected)); + selected.execution = s.execution === "auto" ? best?.execution || "eager" : s.execution; + parallel.push(`--attention-backend ${attention === "sdpa" ? "torch_sdpa" : "fa"}`); + if (selected.execution === "bcg") { + const warmup = { quality: s.quality }; + if (s.mode === "i2i") Object.assign(warmup, { flux3_mode: "i2i", image_path: ["{{WARMUP_IMAGE}}"] }); + parallel.push("--enable-breakable-cuda-graph", "--warmup-mode server", + "--warmup-resolutions 1024x1024", `--warmup-sampling-params '${JSON.stringify(warmup)}'`); + } + const verified = measuredProfiles.some((recipe) => matches(recipe, selected)); const warnings = verified ? [] : ["This exact hardware, checkpoint and topology combination is not yet end-to-end verified."]; + if (verified && s.mode === "i2i") { + warnings.push("Verified recipes cover 1 MP references and output, batch 1; larger inputs require separate capacity validation."); + } + if (selected.execution === "bcg") { + warnings.push(s.mode === "i2i" + ? "Warmup needs a server-local 1024x1024 reference. Longer prompt buckets, other reference sizes/counts, batches or quality settings can fall back to eager." + : "Graph capture reserves extra memory and covers the 80-token text bucket. Longer prompts, different shapes, batches or quality settings can fall back to eager."); + } if (s.hw === "rtx5090" && s.placement === "resident" && Number(s.batch_size) > 1) { warnings.push("Full-resolution batching needs additional activation memory. Use offload or batch 1 if the resident batch does not fit."); } @@ -270,6 +321,8 @@ return { }, resolvedSettings: { placement: s.placement === "resident" ? "GPU resident" : s.placement === "offload" ? "Layerwise offload" : "Auto", + attention: attention === "sdpa" ? "Torch SDPA" : "FlashAttention", + execution: selected.execution === "bcg" ? "Breakable CUDA graphs" : "Eager", }, }, }; @@ -291,6 +344,11 @@ return { label: "I2I input image", default: "/data/flux3/reference.png", }, + WARMUP_IMAGE: { + target: "command", + label: "BCG warmup image (server path)", + default: "/data/flux3/reference.png", + }, }, curl: (s) => { diff --git a/python/sglang/multimodal_gen/configs/pipeline_configs/flux3_image.py b/python/sglang/multimodal_gen/configs/pipeline_configs/flux3_image.py index 77df41f133..297101e5cb 100644 --- a/python/sglang/multimodal_gen/configs/pipeline_configs/flux3_image.py +++ b/python/sglang/multimodal_gen/configs/pipeline_configs/flux3_image.py @@ -94,6 +94,7 @@ class Flux3ImageSamplingParams(SamplingParams): def prepare_synthetic_warmup_request_for_queue( self, req: Any, server_args: Any ) -> None: + self._explicit_fields = {"width", "height"} # Warmup copies the request's steps as the full-workload target before # any adjustment; ``None`` (server default) has to be resolved here. for key in ( diff --git a/python/sglang/multimodal_gen/runtime/layers/attention/backends/sdpa.py b/python/sglang/multimodal_gen/runtime/layers/attention/backends/sdpa.py index c106e537cb..09179ad24a 100644 --- a/python/sglang/multimodal_gen/runtime/layers/attention/backends/sdpa.py +++ b/python/sglang/multimodal_gen/runtime/layers/attention/backends/sdpa.py @@ -35,13 +35,14 @@ _MPS_VARLEN_QUERY_CHUNK_SIZE = 128 def torch_default_sdpa_backends(): - """Context in which SDPA picks from every PyTorch backend, cuDNN included. + """Reference SDPA with a stable cuDNN-first preference and fallback set. The CUDA platform disables cuDNN SDPA process-wide (diffusers#9704). Modules - whose reference outputs were produced with PyTorch's defaults opt back in - around their own ``scaled_dot_product_attention`` calls. + opt back in locally. Priority must be explicit: restoring another SDPA + context can move disabled backends to the end of the ambient priority list, + otherwise changing reference outputs after the first DiT forward. """ - return sdpa_kernel(_PYTORCH_DEFAULT_CUDA_SDP_BACKENDS) + return sdpa_kernel(_PYTORCH_DEFAULT_CUDA_SDP_BACKENDS, set_priority=True) class SDPABackend(AttentionBackend): diff --git a/python/sglang/multimodal_gen/runtime/models/dits/flux3_image.py b/python/sglang/multimodal_gen/runtime/models/dits/flux3_image.py index f07e3bf784..228e7a3ee1 100644 --- a/python/sglang/multimodal_gen/runtime/models/dits/flux3_image.py +++ b/python/sglang/multimodal_gen/runtime/models/dits/flux3_image.py @@ -23,7 +23,7 @@ Native fused kernels are bit-close, not bit-exact, to the torch reference. from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field import torch import torch.nn as nn @@ -96,6 +96,16 @@ _CONTENT_STACK_PREFIX = "content_mode_blocks." TEXT = "text" +@dataclass +class StaticStreams: + """Request-owned work for streams whose tokens and timestep stay fixed.""" + + hidden: dict[str, Tensor] = field(default_factory=dict) + modulation: dict[str, TokenModulation] = field(default_factory=dict) + rope: dict[str, Flux3Rope] = field(default_factory=dict) + joint_rope: Flux3Rope | None = None + + @dataclass class Packed: """Mode-major tokens with per-sample bounds for segmented attention. @@ -114,8 +124,8 @@ class Packed: timesteps_ref: Tensor | None = None # Request-owned positional tables; the ids do not change during denoising. rope: dict[str, Tensor] | None = None - # Eager-only reuse of invariant text/reference mode-block outputs. - static_hidden: dict[str, Tensor] | None = None + # Eager-only reuse of invariant text/reference computation. + static: StaticStreams | None = None @property def num_samples(self) -> int: @@ -133,7 +143,7 @@ class Packed: timesteps = {} mode_indices = {} rope = None if self.rope is None else {} - static_hidden = None if self.static_hidden is None else {} + static = None if self.static is None else StaticStreams() for mode in self.data: bounds = self.mode_indices[mode][first:stop] start, end = bounds[0][0], bounds[-1][1] @@ -144,14 +154,15 @@ class Packed: mode_indices[mode] = [(lo - start, hi - start) for lo, hi in bounds] if rope is not None: rope[mode] = self.rope[mode][start:end] - if static_hidden is not None and mode in self.static_hidden: - static_hidden[mode] = self.static_hidden[mode][start:end] + if static is not None and mode in self.static.hidden: + static.hidden[mode] = self.static.hidden[mode][start:end] + static.modulation[mode] = tuple( + value[start:end] for value in self.static.modulation[mode] + ) timesteps_ref = ( None if self.timesteps_ref is None else self.timesteps_ref[first:stop] ) - return Packed( - data, ids, timesteps, mode_indices, timesteps_ref, rope, static_hidden - ) + return Packed(data, ids, timesteps, mode_indices, timesteps_ref, rope, static) def pack( @@ -797,8 +808,10 @@ class Flux3ImageDiT(Flux3Transformer): def mod_key(mode: str) -> str: return TEXT_STREAM if mode == TEXT else mode - vec = {mode: self._token_vector(timesteps[mode]) for mode in modes} - self._add_reference_time(packed, modes, vec) + cached = {} if packed.static is None else packed.static.hidden + active = tuple(mode for mode in modes if mode not in cached) + vec = {mode: self._token_vector(timesteps[mode]) for mode in active} + self._add_reference_time(packed, active, vec) def broadcast_modulation(values, mode): length = data[mode].shape[0] @@ -813,21 +826,23 @@ class Flux3ImageDiT(Flux3Transformer): mode: broadcast_modulation( self.early_stream_modulations[mod_key(mode)](vec[mode]), mode ) - for mode in modes + for mode in active } single = { mode: broadcast_modulation( self.single_stream_modulations[mod_key(mode)](vec[mode]), mode ) - for mode in modes + for mode in active } + if packed.static is not None: + single.update(packed.static.modulation) x = { mode: ( self.txt_in(data[mode].to(self.dtype)) if mode == TEXT else self.emb_in[mode](data[mode].to(self.dtype)) ) - for mode in modes + for mode in active } rope = { mode: ( @@ -853,9 +868,14 @@ class Flux3ImageDiT(Flux3Transformer): def _packed_blocks(self, packed, content, x, rope, early, single, layout): """All samples in one varlen sequence per block (no sequence parallel).""" - cached = packed.static_hidden if packed.static_hidden is not None else {} + cached = packed.static.hidden if packed.static is not None else {} x.update(cached) - mode_rope = {mode: _with_positions(rows) for mode, rows in rope.items()} + if packed.static is not None and packed.static.rope: + mode_rope = packed.static.rope + else: + mode_rope = {mode: _with_positions(rows) for mode, rows in rope.items()} + if packed.static is not None: + packed.static.rope = mode_rope for depth in range(self.depth): for mode in content: if mode in cached: @@ -873,12 +893,20 @@ class Flux3ImageDiT(Flux3Transformer): early[TEXT], seq_lens=_seq_lens(packed.mode_indices[TEXT]), ) - if packed.static_hidden is not None: - packed.static_hidden.update( + if packed.static is not None: + packed.static.hidden.update( (mode, x[mode]) for mode in (TEXT, "image_cond") if mode in x ) + packed.static.modulation.update( + (mode, single[mode]) for mode in packed.static.hidden + ) hidden = layout.pack(x) - joint_rope = _with_positions(layout.pack(rope)) + if packed.static is not None and packed.static.joint_rope is not None: + joint_rope = packed.static.joint_rope + else: + joint_rope = _with_positions(layout.pack(rope)) + if packed.static is not None: + packed.static.joint_rope = joint_rope joint_mods = tuple( layout.pack({mode: single[mode][index] for mode in single}) for index in range(3) diff --git a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/flux3_image.py b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/flux3_image.py index b3b57a351d..bc066f8eed 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/flux3_image.py +++ b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/flux3_image.py @@ -35,7 +35,12 @@ from sglang.multimodal_gen.runtime.managers.forward_context import ( from sglang.multimodal_gen.runtime.managers.memory_managers.component_manager import ( ComponentUse, ) -from sglang.multimodal_gen.runtime.models.dits.flux3_image import TEXT, Packed, pack +from sglang.multimodal_gen.runtime.models.dits.flux3_image import ( + TEXT, + Packed, + StaticStreams, + pack, +) from sglang.multimodal_gen.runtime.models.vaes.fast_path_gate import use_vae_fast_path from sglang.multimodal_gen.runtime.pipelines.flux3_image_package import ( Flux3ImagePackageConfig, @@ -791,7 +796,7 @@ class Flux3ImageDenoiseStage(DenoisingStage): and batch.skip_softmax_params is None and self.transformer.can_reuse_mode_blocks() ): - packed.static_hidden = {} + packed.static = StaticStreams() self._maybe_enable_cache_dit_and_torch_compile(num_steps, batch) with ( torch.inference_mode(), diff --git a/python/sglang/multimodal_gen/runtime/warmup_request_builder.py b/python/sglang/multimodal_gen/runtime/warmup_request_builder.py index 4076f4755a..e43461c602 100644 --- a/python/sglang/multimodal_gen/runtime/warmup_request_builder.py +++ b/python/sglang/multimodal_gen/runtime/warmup_request_builder.py @@ -629,7 +629,14 @@ def build_warmup_reqs( num_inference_steps=sampling_defaults.num_inference_steps, num_frames=num_frames, ) - if include_warmup_image: + if ( + server_args.enable_breakable_cuda_graph is True + and sampling_defaults.image_path + ): + # Optional image conditioning changes the captured signature. Use + # the configured references rather than capturing text-only inputs. + req_kwargs["image_path"] = sampling_defaults.image_path + elif include_warmup_image: if warmup_input_path is None: raise RuntimeError( "Warmup image path is required for image-input model" diff --git a/python/sglang/multimodal_gen/test/server/test_server_flux3_image.py b/python/sglang/multimodal_gen/test/server/test_server_flux3_image.py index 11cc3a2b43..d52725dc44 100644 --- a/python/sglang/multimodal_gen/test/server/test_server_flux3_image.py +++ b/python/sglang/multimodal_gen/test/server/test_server_flux3_image.py @@ -87,6 +87,7 @@ def test_grouped_requests_multi_output_and_repeat(diffusion_server): for editing in (False, True): serial = [generate(index, editing)[0] for index in range(2)] + np.testing.assert_array_equal(generate(editing=editing)[0], serial[0]) barrier = Barrier(2) with ThreadPoolExecutor(2) as pool: grouped = list( diff --git a/python/sglang/multimodal_gen/test/unit/test_cfg_parallel_warmup.py b/python/sglang/multimodal_gen/test/unit/test_cfg_parallel_warmup.py index 3dd3fa1b6f..5fed0f3a02 100644 --- a/python/sglang/multimodal_gen/test/unit/test_cfg_parallel_warmup.py +++ b/python/sglang/multimodal_gen/test/unit/test_cfg_parallel_warmup.py @@ -22,6 +22,9 @@ from unittest.mock import MagicMock, patch import torch from sglang.multimodal_gen.configs.pipeline_configs.base import ModelTaskType +from sglang.multimodal_gen.configs.pipeline_configs.flux3_image import ( + Flux3ImageSamplingParams, +) from sglang.multimodal_gen.configs.pipeline_configs.flux_finetuned import ( Flux2FinetunedPipelineConfig, ) @@ -1062,6 +1065,35 @@ class TestWarmupReqCfgParallel(unittest.TestCase): ["/tmp/warmup.png", "/tmp/warmup.png"], ) + def test_breakable_warmup_uses_configured_references(self): + server_args = MagicMock() + server_args.warmup_steps = 1 + server_args.enable_cfg_parallel = False + server_args.enable_torch_compile = False + server_args.enable_breakable_cuda_graph = True + server_args.pipeline_config.task_type = ModelTaskType.TI2I + references = ["first.png", "second.png"] + + with patch( + "sglang.multimodal_gen.runtime.warmup_request_builder.get_model_sampling_defaults", + return_value=Flux3ImageSamplingParams( + width=512, + height=512, + image_path=references, + flux3_mode="i2i", + num_inference_steps=16, + ), + ): + req = build_warmup_reqs( + server_args, + warmup_resolutions=["512x512"], + server_based_warmup=True, + )[0] + + self.assertEqual(req.image_path, references) + self.assertEqual(req.flux3_mode, "i2i") + self.assertEqual(req.sampling_params._explicit_fields, {"width", "height"}) + def test_server_based_warmup_keeps_required_image_input(self): server_args = MagicMock() server_args.warmup_steps = 1 diff --git a/python/sglang/multimodal_gen/test/unit/test_flux3_image_dit.py b/python/sglang/multimodal_gen/test/unit/test_flux3_image_dit.py index 3411689b38..1d023c93cb 100644 --- a/python/sglang/multimodal_gen/test/unit/test_flux3_image_dit.py +++ b/python/sglang/multimodal_gen/test/unit/test_flux3_image_dit.py @@ -198,10 +198,18 @@ def test_packed_cfg_forward_matches_flux3_transformer_per_sample( packed.rope = {mode: native.rope(ids) for mode, ids in packed.ids.items()} cached = native(packed)["image"].float() torch.testing.assert_close(cached, predicted, rtol=0, atol=0) - packed.static_hidden = {} - for _ in range(2): - reused = native(packed)["image"].float() - torch.testing.assert_close(reused, predicted, rtol=0, atol=0) + packed.static = flux3_image.StaticStreams() + modulation = native.single_stream_modulations["txt"] + with ( + mock.patch.object( + native.txt_in, "forward", wraps=native.txt_in.forward + ) as embed, + mock.patch.object(modulation, "forward", wraps=modulation.forward) as mod, + ): + for _ in range(2): + reused = native(packed)["image"].float() + torch.testing.assert_close(reused, predicted, rtol=0, atol=0) + assert embed.call_count == mod.call_count == 1 for mode, rows in packed.sample(1).rope.items(): start, end = packed.mode_indices[mode][1] assert torch.equal(rows, packed.rope[mode][start:end]) @@ -231,7 +239,7 @@ def test_packed_cfg_forward_matches_flux3_transformer_per_sample( packed.timesteps["image"].fill_(0.3) packed.data["image"].mul_(0.9) reused = native(packed)["image"] - packed.static_hidden = None + packed.static = None uncached = native(packed)["image"] torch.testing.assert_close(reused, uncached, rtol=0, atol=0) diff --git a/python/sglang/multimodal_gen/test/unit/test_sdpa_cudnn_priority.py b/python/sglang/multimodal_gen/test/unit/test_sdpa_cudnn_priority.py index 5e7ebba1ef..c8b2c8720d 100644 --- a/python/sglang/multimodal_gen/test/unit/test_sdpa_cudnn_priority.py +++ b/python/sglang/multimodal_gen/test/unit/test_sdpa_cudnn_priority.py @@ -21,6 +21,7 @@ from torch.nn.attention import SDPBackend, sdpa_kernel from sglang.multimodal_gen.runtime.layers.attention.backends.sdpa import ( _PYTORCH_DEFAULT_CUDA_SDP_BACKENDS, SDPAImpl, + torch_default_sdpa_backends, ) NUM_HEADS, HEAD_DIM, SEQ_LEN = 8, 128, 512 @@ -92,3 +93,19 @@ def test_opting_out_leaves_backend_selection_alone(): def test_cpu_tensors_do_not_get_a_cuda_context(): cpu_q = torch.randn(1, NUM_HEADS, 16, HEAD_DIM) assert isinstance(_impl(True)._sdpa_context(cpu_q), type(nullcontext())) + + +def test_reference_sdpa_does_not_inherit_a_previous_attention_priority(): + q, k, v = _qkv() + with sdpa_kernel(_PYTORCH_DEFAULT_CUDA_SDP_BACKENDS, set_priority=True): + with sdpa_kernel(SDPBackend.CUDNN_ATTENTION): + try: + F.scaled_dot_product_attention(q, k, v) + except RuntimeError: + pytest.skip("no cuDNN attention kernel for this shape") + for priority in ( + _PYTORCH_DEFAULT_CUDA_SDP_BACKENDS, + list(reversed(_PYTORCH_DEFAULT_CUDA_SDP_BACKENDS)), + ): + with sdpa_kernel(priority, set_priority=True): + assert _kernel_under(torch_default_sdpa_backends(), q, k, v) == "cudnn"