diff --git a/.claude/skills/cookbook-add-model/SKILL.md b/.claude/skills/cookbook-add-model/SKILL.md index 1cd765901..c0d2e5988 100644 --- a/.claude/skills/cookbook-add-model/SKILL.md +++ b/.claude/skills/cookbook-add-model/SKILL.md @@ -32,6 +32,7 @@ lacks, and replace the EXAMPLE cells with verified recipes. DeepSeek-V4 is a pop - [references/mintlify-authoring.md](references/mintlify-authoring.md) — MDX rules (forbidden syntax, JSX tables, labeled fences) + invocation-example patterns. Read before writing §1–§3 prose. - [references/engine-axis.md](references/engine-axis.md) — adding a new Playground feature axis (rare engine work). - [references/vendor-logo.md](references/vendor-logo.md) — new-vendor card logo: ask the user for the brand logo, then generate the icon-only 940×525 RGBA PNG (spec + Pillow recipe + `git add -f`). +- [references/diffusion-authoring.md](references/diffusion-authoring.md) — required opening, tag, command-picker, and feature-overlay contract for diffusion model pages. Read before editing any `docs/cookbook/diffusion//.mdx` page. ## Architecture at a glance @@ -68,9 +69,12 @@ playground reads it), the **`sglang-deploy-sel` custom event** (deploy dispatche every change; playground listens — `replaceState` doesn't fire `hashchange`), and the shared **`sglang-deploy-env` localStorage key** (HOST/PORT placeholders). -> The template is **autoregressive**. Diffusion / omni pages follow their own category -> structure — don't force the config-driven template on them; still obey the Mintlify / -> NEW-tag / docs.json / category-card / validation rules below. +> The main template is **autoregressive**. Diffusion pages use +> `templates/diffusion-page.mdx.tmpl` plus +> [references/diffusion-authoring.md](references/diffusion-authoring.md); do not force the +> autoregressive deployment matrix on them. Omni pages follow their own category structure. +> All categories still obey the Mintlify / NEW-tag / docs.json / category-card / validation +> rules below. --- @@ -131,6 +135,15 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h ## Phase 2 — Instantiate the template +For a diffusion model, instantiate `templates/diffusion-page.mdx.tmpl` and keep the +shared `DiffusionModelTags` component, plus `templates/diffusion-config.jsx.tmpl` for the +opt-in scoped command builder. Put the compact install command and builder in §1 Quick +start. The first two paragraphs in §2 Model capabilities are not generic filler: they must +state the model's capability range, strongest differentiator, when to choose it, and at +least one real deployment or capability boundary. Put orthogonal runtime features in +`scope: "serve"` or `scope: "request"`, not in the base recipe; use the schema from +`references/diffusion-authoring.md`. + 1. **Copy** the three template files to their target paths (above). Note the two vendor-folder conventions: under `configs/` the folder is the **HuggingFace org** (`deepseek-ai`); under `cookbook/` it's the **display vendor** (`DeepSeek`). @@ -206,6 +219,10 @@ user notes: §1 Model Introduction (description, links, params, license, variant Tool-Calling / HiCache — keep only what applies; match the reasoning example to the parser's output shape; each runnable block gets an `**Output Example:**`). +For diffusion pages, follow the category-specific Quick start and capability contract in +`references/diffusion-authoring.md`. Run `node docs/scripts/check_cookbook_configs.mjs` to +verify the tag widget and introduction structure before rendering the page. + ## Phase 6 — Review ``` diff --git a/.claude/skills/cookbook-add-model/references/diffusion-authoring.md b/.claude/skills/cookbook-add-model/references/diffusion-authoring.md new file mode 100644 index 000000000..3c2065dc1 --- /dev/null +++ b/.claude/skills/cookbook-add-model/references/diffusion-authoring.md @@ -0,0 +1,99 @@ +# Diffusion cookbook authoring contract + +Use this contract for every model page under +`docs/cookbook/diffusion//.mdx`. + +## Opening structure + +Import and render the shared tag widget immediately after frontmatter: + +```mdx +import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; + + +``` + +Use 4–6 tags, ordered from broad to specific: + +1. output modality or product class; +2. primary request modes; +3. the capability that differentiates the model; +4. an important scale, latency, duration, resolution, or architecture fact. + +Do not spend tags on generic claims such as `native`, `fast`, `high quality`, or +`SGLang`. Attention backends, Cache-DiT, and online quantization are feature overlays, +not model identity, unless a published checkpoint is intrinsically tied to that format. + +The page must start with `## 1. Quick start`: keep installation to one runnable +command, add at most one short orientation paragraph, and render the scoped command +builder before detailed capability tables or deployment commentary. A reader should +reach a generated Serve command without scrolling through model background first. + +Follow it with `## 2. Model capabilities` and 2–3 short paragraphs: + +- paragraph 1: what the model does and where it is genuinely strong; +- paragraph 2: when to choose it and the most important limitation or tradeoff; +- optional paragraph/table: checkpoint or mode routing that a user must understand before serving. + +Keep the capability lead between roughly 45 and 180 words. Replace marketing superlatives with +concrete capabilities. A reader should learn more than “this is a powerful image/video +model,” but should not have to read the architecture section to choose the right model. + +## Scoped command builder + +New diffusion pages use `templates/diffusion-config.jsx.tmpl` and opt into the shared +`commandBuilder` renderer. Setup, Server, and Request choices share one semantic selection and +one command composer; do not create a second command engine or assemble fragments in MDX. + +Classify every dimension by lifecycle: + +- `scope: "base"`: the visible Setup tab—hardware-independent required decisions such as + checkpoint weights and request mode. Hardware, Nodes × GPUs/node, and the recommended + verified deployment are supplied by the shared builder. +- `scope: "serve"`: startup flags such as placement, attention, precision, encoder scheduling, + and graph execution. They modify only the complete Serve command. +- `scope: "request"`: sampling fields such as quality and outputs. They modify only the complete + Request command. + +Keep the topology registry small and honest. `verifiedRecipes` contains only exact end-to-end +runs on that hardware and resource shape. `autoTopology(selection)` may construct a legal custom +shape, but `resolveDeployment` must mark it `unverified` unless it exactly matches a recipe. +`validateTopology` returns static errors for impossible world-size, head, partition, or placement +combinations; errors disable both Copy actions. Never silently label a nearby GPU or topology as +verified. + +Expose topology-dependent best values explicitly. A Server row may remain `Auto`, but its summary +and generated flag must show the effective policy—for example, encoder `auto` resolving to +`replicate` across nodes. Reserve `disabled` for combinations that cannot work (a kernel the +platform does not ship, a mode the checkpoint cannot serve); an option that runs but has not +been through a verification round should declare `soft` instead, which keeps it selectable and +labels the pick as unverified. Keep `torch.compile` and similarly narrow experiments in the +detailed prose until they have a broadly compatible recipe. + +Each dimension should provide a concise `description`, an optional quality label, and a +`learnMore` anchor. Each option should provide at most two lines of decision-relevant explanation, +its exact `flags`/`stripPrefixes`/`env`/`hints`, and a `disabled` (with `disableReason`), `soft` +(with `softReason`), or `verifiedWhen` predicate when support is conditional — the reason strings +are user-facing: blocked options flash theirs under the row on tap and expose it as a tooltip. +The builder stores all semantic choices in the URL hash; active scope, expanded state, head +address, and node rank stay local. + +Legacy configs without `commandBuilder` continue to use the old matrix renderer. Do not migrate +an existing page opportunistically; use the new schema for new diffusion models and deliberate +model-by-model migrations. + +## Review checklist + +- tags render before section 1 and describe the model rather than the runtime; +- Quick start puts installation and the generated commands before detailed model background; +- the capability section explains strength and boundary without marketing filler; +- checkpoint variants and request modes are unambiguous; +- Setup keeps both commands visible as a deployment overview; Server and Request each show only + the command controlled by that scope; +- command bodies grow naturally up to their collapsed limit and do not reserve empty height; +- topology-dependent recommended defaults are explicit in the setting summary and generated command; +- legal custom topologies are Unverified and copyable; statically illegal combinations block Copy; +- attention, quantization, caching, compile, and similar orthogonal features have explicit + quality contracts and verified scopes; +- unverified hardware or performance claims are absent; +- `node docs/scripts/check_cookbook_configs.mjs` and Mintlify validation pass. diff --git a/.claude/skills/cookbook-add-model/templates/diffusion-config.jsx.tmpl b/.claude/skills/cookbook-add-model/templates/diffusion-config.jsx.tmpl new file mode 100644 index 000000000..9f0da490e --- /dev/null +++ b/.claude/skills/cookbook-add-model/templates/diffusion-config.jsx.tmpl @@ -0,0 +1,173 @@ +// Diffusion command-builder config template. Replace every __TOKEN__, keep only +// verified hardware/features, and adapt the topology divisibility contract to +// the model architecture. + +export const config = { + modelName: "__MODEL_DISPLAY__", + supportedHardware: ["__DEFAULT_HW__"], + groupHardware: false, + matchDims: [], + + overlayDims: [ + { + id: "weights", + title: "Checkpoint weights", + scope: "base", + description: "Choose the checkpoint partition required by this request mode.", + default: "default", + options: [{ id: "default", label: "Default", flags: [] }], + }, + { + id: "mode", + title: "Request mode", + scope: "base", + default: "text", + options: [{ id: "text", label: "Text" }], + }, + { + id: "placement", + title: "Placement", + scope: "serve", + description: "Keep weights resident unless a verified capacity path requires sharding or offload.", + default: "resident", + options: [ + { id: "resident", label: "Resident", flags: ["--performance-mode speed"], recommended: true }, + { id: "fsdp", label: "FSDP", flags: ["--use-fsdp-inference true"], disabled: true }, + { id: "offload", label: "Layerwise offload", disabled: true }, + ], + }, + { + id: "attention", + title: "Attention", + scope: "serve", + description: "Use the platform default unless another backend was measured end to end.", + default: "platform", + options: [{ id: "platform", label: "Platform default", recommended: true }], + }, + { + id: "precision", + title: "Precision", + scope: "serve", + default: "native", + options: [{ id: "native", label: "Native mixed precision", recommended: true }], + }, + { + id: "encoder", + title: "Encoder", + scope: "serve", + default: "auto", + options: [{ id: "auto", label: "Auto", flags: ["--encoder-parallel auto"], recommended: true }], + }, + { + id: "execution", + title: "Execution", + scope: "serve", + default: "eager", + options: [{ id: "eager", label: "Eager", recommended: true }], + }, + { + id: "quality", + title: "Quality", + scope: "request", + default: "lossless", + options: [{ id: "lossless", label: "Lossless", recommended: true }], + }, + { + id: "outputs", + title: "Outputs", + scope: "request", + kind: "number", + min: 1, + max: 10, + unit: "outputs per prompt", + default: 1, + options: [], + }, + ], + + commandBuilder: { + defaultSelection: { + hw: "__DEFAULT_HW__", + nodes: 1, + gpus_per_node: __DEFAULT_GPUS__, + topology_mode: "auto", + tp_size: 1, + ulysses_degree: __DEFAULT_GPUS__, + ring_degree: 1, + }, + resource: { + limits: { + nodes: { min: 1, max: __MAX_NODES__ }, + gpus_per_node: { min: 1, max: __MAX_GPUS_PER_NODE__ }, + }, + verifiedRecipes: [{ + id: "__DEFAULT_HW__-default", + hw: "__DEFAULT_HW__", + nodes: 1, + gpus_per_node: __DEFAULT_GPUS__, + placement: "resident", + tp_size: 1, + ulysses_degree: __DEFAULT_GPUS__, + ring_degree: 1, + encoder: "auto", + default: true, + }], + autoTopology: (s) => ({ + tp_size: 1, + ulysses_degree: Number(s.gpus_per_node), + ring_degree: Number(s.nodes), + }), + validateTopology: (s, topology) => { + const world = Number(s.nodes) * Number(s.gpus_per_node); + const product = Number(topology.tp_size) * Number(topology.ulysses_degree) * Number(topology.ring_degree); + return world === product ? [] : [`World size ${world} must equal TP × Ulysses × Ring (${product}).`]; + }, + }, + resolveDeployment: (s) => { + const resource = config.commandBuilder.resource; + const topology = s.topology_mode === "manual" + ? { tp_size: Number(s.tp_size), ulysses_degree: Number(s.ulysses_degree), ring_degree: Number(s.ring_degree) } + : resource.autoTopology(s); + const errors = resource.validateTopology(s, topology); + const recipe = resource.verifiedRecipes.find((entry) => entry.hw === s.hw + && entry.nodes === Number(s.nodes) + && entry.gpus_per_node === Number(s.gpus_per_node) + && entry.placement === s.placement + && entry.tp_size === topology.tp_size + && entry.ulysses_degree === topology.ulysses_degree + && entry.ring_degree === topology.ring_degree); + const world = Number(s.nodes) * Number(s.gpus_per_node); + const flags = ["--model-path {{MODEL_NAME}}", `--num-gpus ${world}`]; + if (topology.tp_size > 1) flags.push(`--tp-size ${topology.tp_size}`); + flags.push(`--ulysses-degree ${topology.ulysses_degree}`); + if (topology.ring_degree > 1) flags.push(`--ring-degree ${topology.ring_degree}`); + flags.push("--host {{HOST_IP}}", "--port {{PORT}}"); + const verified = !!recipe && errors.length === 0; + return { + match: { hw: s.hw }, + nnodes: Number(s.nodes), + verified, + flags, + builder: { + topology, + topologySummary: `TP ${topology.tp_size} · Ulysses ${topology.ulysses_degree} · Ring ${topology.ring_degree}`, + errors, + warnings: verified ? [] : ["Valid custom topology; exact end-to-end verification is pending."], + verification: { serve: verified ? "verified" : "unverified", request: verified ? "verified" : "unverified" }, + }, + }; + }, + }, + + modelNames: { default: "__MODEL_ID__" }, + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30010" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30010" }, + }, + curl: (s) => `curl -sS -X POST http://{{CURL_HOST}}:{{CURL_PORT}}/__REQUEST_PATH__ \\ + -H 'Content-Type: application/json' \\ + -d '${JSON.stringify({ model: "{{MODEL_NAME}}", prompt: "__PROMPT__", quality: s.quality, num_outputs_per_prompt: Number(s.outputs) }, null, 2)}'`, + cells: [], +}; diff --git a/.claude/skills/cookbook-add-model/templates/diffusion-page.mdx.tmpl b/.claude/skills/cookbook-add-model/templates/diffusion-page.mdx.tmpl new file mode 100644 index 000000000..ab3f04e1c --- /dev/null +++ b/.claude/skills/cookbook-add-model/templates/diffusion-page.mdx.tmpl @@ -0,0 +1,44 @@ +--- +title: __MODEL_DISPLAY__ +description: "Deploy __MODEL_DISPLAY__ with SGLang Diffusion." +metatags: + description: "Serve __MODEL_DISPLAY__ for __PRIMARY_CAPABILITY__ with SGLang Diffusion." +--- + +import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; +import { Deployment } from '/src/snippets/_deployment.jsx'; +import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx'; + + + +## 1. Quick start + +Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose +a verified recipe below. + + + +## 2. Model capabilities + +[__MODEL_DISPLAY__](__MODEL_URL__) is __CONCRETE_MODEL_DESCRIPTION__. Its strongest use case is __MEASURED_OR_DOCUMENTED_STRENGTH__. + +Choose it when __SELECTION_GUIDANCE__. The main tradeoff is __CAPABILITY_OR_DEPLOYMENT_BOUNDARY__. + +| Checkpoint or mode | Best fit | Important boundary | +| --- | --- | --- | +| `__MODEL_ID__` | __USE_CASE__ | __LIMIT__ | + +## 3. Deployment details + +Document topology-dependent defaults, platform-specific installation details, +and only the hardware recipes that need explanation beyond the builder. + +## 4. Request examples + +__REQUEST_EXAMPLES__ + +## 5. Feature details + +Document quality contracts, installation requirements, and benchmark scope for +the Server and Request choices exposed in the builder. Keep `torch.compile` and +other experimental controls here until they have a broadly compatible recipe. diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index f1677f0ff..c4bd0feda 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -80,7 +80,7 @@ jobs: with: node-version: "20" - - name: Check cookbook configs + - name: Check cookbook authoring contracts run: node docs/scripts/check_cookbook_configs.mjs # Hard gate: docs internal links/anchors/redirects must resolve. diff --git a/docs/cookbook/diffusion/Cosmos/Cosmos3.mdx b/docs/cookbook/diffusion/Cosmos/Cosmos3.mdx index 3b9c70a13..6fd48027f 100644 --- a/docs/cookbook/diffusion/Cosmos/Cosmos3.mdx +++ b/docs/cookbook/diffusion/Cosmos/Cosmos3.mdx @@ -6,11 +6,13 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + ## 1. Model Introduction -[NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is an omnimodal world-model family for image, video, sound, and action generation. SGLang Diffusion serves the public checkpoints with its native Cosmos3 pipeline. +[NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is an omnimodal world-model family spanning text/image/video generation, optional synchronized sound, and robot action prediction. Its main advantage is breadth: the same native SGLang pipeline can serve media-generation checkpoints and the DROID policy checkpoint without routing through an LLM sampler. + +Choose Nano for the broadest modality coverage and lower deployment cost, Super for the larger 64B image/video model, and a specialized checkpoint when only T2I or I2V is needed. Sound and action are checkpoint-specific heads, so they are not available from every Cosmos3 repository. | Model | Status | Notes | | --- | --- | --- | diff --git a/docs/cookbook/diffusion/Ernie-Image/Ernie-Image.mdx b/docs/cookbook/diffusion/Ernie-Image/Ernie-Image.mdx index 0d6df4e51..a4c420a07 100644 --- a/docs/cookbook/diffusion/Ernie-Image/Ernie-Image.mdx +++ b/docs/cookbook/diffusion/Ernie-Image/Ernie-Image.mdx @@ -6,11 +6,13 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + -## 1. Model introduction +## 1. Model Introduction -[ERNIE-Image](https://huggingface.co/baidu/ERNIE-Image) is Baidu's text-to-image diffusion model family. SGLang Diffusion supports both the regular and Turbo checkpoints with the native `ErnieImagePipeline`. +[ERNIE-Image](https://huggingface.co/baidu/ERNIE-Image) is Baidu's text-to-image family with separate standard and Turbo checkpoints. The standard model is the quality-oriented path; Turbo is the lower-latency choice, and both load through SGLang's native `ErnieImagePipeline`. + +This integration currently targets text-only image generation rather than image editing or reference conditioning. Choose it for a straightforward single-GPU T2I deployment; use a dedicated editing model when preserving source-image structure is part of the task. | Model | Hugging Face model ID | Notes | | --- | --- | --- | diff --git a/docs/cookbook/diffusion/FLUX/FLUX.mdx b/docs/cookbook/diffusion/FLUX/FLUX.mdx index 24bc85a0d..05c0c4deb 100644 --- a/docs/cookbook/diffusion/FLUX/FLUX.mdx +++ b/docs/cookbook/diffusion/FLUX/FLUX.mdx @@ -7,31 +7,18 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; import { FluxDeployment } from '/src/snippets/diffusion/flux-deployment.jsx'; - + ## 1. Model Introduction -[FLUX](https://blackforestlabs.ai/) is a family of rectified flow transformer models developed by Black Forest Labs for high-quality image generation from text descriptions. +[FLUX](https://blackforestlabs.ai/) is Black Forest Labs' rectified-flow image model family. [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) is the smaller 12B text-to-image checkpoint; [FLUX.2-dev](https://huggingface.co/black-forest-labs/FLUX.2-dev) is a 32B model that adds instruction-based editing plus single- and multi-reference composition. -[FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions. +FLUX is a strong default when prompt adherence, polished image quality, or reference consistency matters. The tradeoff is deployment weight: FLUX.2 needs substantially more memory than FLUX.1, and the dev checkpoints use the FLUX non-commercial license, so review the model license before production use. -**Key Features:** - -- **Cutting-edge Output Quality**: Second only to the state-of-the-art FLUX.1 [pro] model -- **Competitive Prompt Following**: Matches the performance of closed-source alternatives -- **Guidance Distillation**: Trained using guidance distillation for improved efficiency -- **Open Weights**: Available for personal, scientific, and commercial purposes under the FLUX [dev] Non-Commercial License - -[FLUX.2-dev](https://huggingface.co/black-forest-labs/FLUX.2-dev) is a 32 billion parameter rectified flow transformer capable of generating, editing, and combining images based on text instructions. - -**Key Features:** - -- **State-of-the-art Performance**: Leading open model in text-to-image generation, single-reference editing, and multi-reference editing -- **No Finetuning Required**: Character, object, and style reference without additional training in one model -- **Guidance Distillation**: Trained using guidance distillation for improved efficiency -- **Open Weights**: Available for personal, scientific, and commercial purposes under the FLUX [dev] Non-Commercial License - -For more details, please refer to the [FLUX.1-dev HuggingFace page](https://huggingface.co/black-forest-labs/FLUX.1-dev), [FLUX.2-dev HuggingFace page](https://huggingface.co/black-forest-labs/FLUX.2-dev), and the [official blog post](https://blackforestlabs.ai/announcing-black-forest-labs/). +| Checkpoint | Best fit | Main limitation | +| --- | --- | --- | +| `black-forest-labs/FLUX.1-dev` | High-quality text-to-image with the lighter FLUX deployment | No native multi-reference editing path | +| `black-forest-labs/FLUX.2-dev` | Text-to-image, editing, and reference-guided composition in one model | 32B model with a larger memory footprint | ## 2. SGLang-diffusion Installation diff --git a/docs/cookbook/diffusion/Ideogram/Ideogram4.mdx b/docs/cookbook/diffusion/Ideogram/Ideogram4.mdx index a03d32dea..c02530e6a 100644 --- a/docs/cookbook/diffusion/Ideogram/Ideogram4.mdx +++ b/docs/cookbook/diffusion/Ideogram/Ideogram4.mdx @@ -6,13 +6,13 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + -## 1. Model introduction +## 1. Model Introduction -[Ideogram 4](https://huggingface.co/ideogram-ai/ideogram-4-nf4) is Ideogram's text-to-image diffusion model. SGLang Diffusion supports the official NF4 and FP8 checkpoints, the Comfy-Org NVFP4 transformer checkpoint, and fal's single-branch Fast and Instant variants. +[Ideogram 4](https://huggingface.co/ideogram-ai/ideogram-4-nf4) is a text-to-image family focused on polished composition and typography-aware generation. SGLang supports the official NF4 and FP8 checkpoints, Comfy-Org's NVFP4 transformer, and fal's distilled Fast and Instant variants. -Compared with previous open-source image models, Ideogram 4 provides a significant aesthetic lift, with stronger composition, more polished visual style, and better typography-aware generation. +Choose NF4 when memory is the primary constraint, FP8/NVFP4 for supported accelerator deployments, Fast for a 20-step balance, and Instant for the 8-step latency target. The public repositories are gated, and the available releases are already quantized or distilled rather than a single full-precision baseline. | Variant | Hugging Face model ID | Notes | | --- | --- | --- | diff --git a/docs/cookbook/diffusion/JoyEcho/JoyEcho.mdx b/docs/cookbook/diffusion/JoyEcho/JoyEcho.mdx index b06b67517..e9ab40e44 100644 --- a/docs/cookbook/diffusion/JoyEcho/JoyEcho.mdx +++ b/docs/cookbook/diffusion/JoyEcho/JoyEcho.mdx @@ -7,13 +7,15 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + ## 1. Model Introduction -[JoyAI-Echo](https://huggingface.co/jdopensource/JoyAI-Echo) (JoyEcho) is a long-form audio–video generation model built on the LTX-2 backbone. Its core idea is a **paired audio–video memory bank**: each shot commits decoded frames and audio latents into a rolling bank, and subsequent shots condition on that memory prefix. This enables **multi-shot, minute-scale generation** with visual and audio continuity across prompts. +[JoyAI-Echo](https://huggingface.co/jdopensource/JoyAI-Echo) is an 8-step long-form audio-video model built on LTX-2. Its paired memory bank carries decoded visual context and audio latents across prompt changes, making it strongest for multi-shot, minute-scale sequences that need continuity in both picture and soundtrack. -Use `jdopensource/JoyAI-Echo` as `--model-path`. SGLang loads the monolithic release through the built-in [JoyAI-Echo-overlay](https://huggingface.co/Niehen6174/JoyAI-Echo-overlay) materialization path, similar to LTX-2.3-overlay. +Choose JoyEcho over a standard LTX pipeline when shots must share audiovisual memory. Its distilled 832×480 path prioritizes long-form continuity and throughput rather than the higher-resolution two-stage quality modes offered by LTX-2.3. + +Use `jdopensource/JoyAI-Echo` as `--model-path`; SGLang materializes the monolithic release through the built-in [JoyAI-Echo overlay](https://huggingface.co/Niehen6174/JoyAI-Echo-overlay). | Aspect | Standard LTX-2.3 | JoyEcho | | --- | --- | --- | diff --git a/docs/cookbook/diffusion/Krea/Krea-2.mdx b/docs/cookbook/diffusion/Krea/Krea-2.mdx index 636d917f4..725d2dc1f 100644 --- a/docs/cookbook/diffusion/Krea/Krea-2.mdx +++ b/docs/cookbook/diffusion/Krea/Krea-2.mdx @@ -6,24 +6,18 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + ## 1. Model Introduction -[Krea-2](https://huggingface.co/krea/Krea-2-Turbo) is a high-quality text-to-image diffusion model from [Krea](https://www.krea.ai/). It ships in two variants that share the same backbone and differ only in their sampling recipe: +[Krea-2](https://huggingface.co/krea/Krea-2-Turbo) is Krea's photorealistic text-to-image family, built as a single-stream MMDiT with a Qwen3-VL text encoder and Qwen-Image VAE. Both public variants use the same native SGLang pipeline and differ mainly in their sampling target. -- **[Krea-2-Turbo](https://huggingface.co/krea/Krea-2-Turbo)** - a distilled, few-step model that produces photorealistic images in only **8 inference steps** with no classifier-free guidance (`guidance_scale = 1.0`), ideal for fast and interactive generation. -- **[Krea-2-Raw](https://huggingface.co/krea/Krea-2-Raw)** - the base (non-distilled) model that trades speed for maximum fidelity, using a longer schedule (~52 steps) with classifier-free guidance (`guidance_scale ≈ 4.5`). +Choose Turbo for interactive generation: it is distilled to 8 steps with `guidance_scale=1.0`. Choose Raw when maximum fidelity matters more than latency: it uses roughly 52 steps with classifier-free guidance. Neither checkpoint is an image-editing model; use the Qwen-Image-Edit or FLUX.2 path when an input image must be preserved or transformed. -Both variants are built on a single-stream MMDiT with a Qwen3-VL text encoder and the Qwen-Image VAE, and are distributed in the standard diffusers layout (a `model_index.json` plus sharded `transformer/`, `text_encoder/`, `vae/`, `tokenizer/`, and `scheduler/` folders). SGLang loads them **natively** - just point `--model-path` at the repo, no conversion step required. - -**Key Features:** - -- **Two variants, one pipeline**: switch between fast (Turbo) and high-fidelity (Raw) by changing only the model path and the sampling settings. -- **Photorealistic generation** at 1024x1024 and other resolutions. -- **Native diffusers loading**: components (DiT, text encoder, VAE, scheduler) are read straight from the repo's `model_index.json`. - -For more details, see the [Krea-2-Turbo](https://huggingface.co/krea/Krea-2-Turbo) and [Krea-2-Raw](https://huggingface.co/krea/Krea-2-Raw) HuggingFace pages. +| Variant | Model ID | Sampling profile | +| --- | --- | --- | +| Turbo | `krea/Krea-2-Turbo` | 8 steps, no CFG; fastest path | +| Raw | `krea/Krea-2-Raw` | About 52 steps with CFG; higher-fidelity path | ## 2. SGLang-diffusion Installation diff --git a/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx b/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx index 35b9bd804..56f22bafe 100644 --- a/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx +++ b/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx @@ -8,13 +8,15 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; import { LTXDeployment } from '/src/snippets/diffusion/ltx-deployment.jsx'; - + ## 1. Model Introduction -[LTX-2](https://huggingface.co/Lightricks/LTX-2) and [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) are video generation models from Lightricks. SGLang Diffusion supports the LTX series through native one-stage and two-stage pipelines for text-to-video and image-conditioned video generation. +[LTX-2](https://huggingface.co/Lightricks/LTX-2) and [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) are Lightricks video models for text-to-video and image-conditioned generation. Their defining deployment choice is the pipeline: one-stage minimizes latency, standard two-stage refines quality, and LTX-2.3 adds a 1920×1088 HQ path. -Use `Lightricks/LTX-2` or `Lightricks/LTX-2.3` as `--model-path`. For two-stage generation, SGLang uses the spatial upsampler and distilled LoRA components from the model snapshot by default. LTX-2.3 also supports the HQ two-stage variant. +Choose LTX when one model family must span fast generation, higher-quality refinement, and LoRA adaptation. Two-stage quality costs substantially more memory and startup work because both DiTs and the spatial upsampler participate; use the one-stage path when latency or capacity matters more than final detail. + +Use `Lightricks/LTX-2` or `Lightricks/LTX-2.3` as `--model-path`. SGLang loads the two-stage upsampler and distilled components from the selected snapshot. **License notice:** LTX-2 and LTX-2.3 are released under the LTX-2 Community License Agreement, not Apache 2.0. The license includes commercial-use restrictions for some entities. Review the [official Lightricks license](https://huggingface.co/Lightricks/LTX-2.3/blob/main/LICENSE) before production or commercial use; SGLang support does not grant additional model usage rights. diff --git a/docs/cookbook/diffusion/LTX/LTX2.5.mdx b/docs/cookbook/diffusion/LTX/LTX2.5.mdx index d1ced3fdd..69ff7e031 100644 --- a/docs/cookbook/diffusion/LTX/LTX2.5.mdx +++ b/docs/cookbook/diffusion/LTX/LTX2.5.mdx @@ -8,7 +8,7 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; import { LTX25Deployment } from '/src/snippets/diffusion/ltx25-deployment.jsx'; - + ## 1. Model Introduction diff --git a/docs/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE.mdx b/docs/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE.mdx index f34043b6a..21ffabbaf 100644 --- a/docs/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE.mdx +++ b/docs/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE.mdx @@ -7,13 +7,22 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + -## 1. Model introduction +## 1. Model Introduction [LingBot Video MoE 30B-A3B](https://huggingface.co/robbyant/lingbot-video-moe-30b-a3b) -is a text-to-video mixture-of-experts model. SGLang Diffusion provides a native -pipeline for the public checkpoint: +is a text-to-video mixture-of-experts model served through a native SGLang +Diffusion pipeline. Its main advantage is the MoE economics: 30B total +parameters with 3B active per token, so a plain `sglang serve` invocation +covers the released 480×480, 81-frame profile without parallelism flags. + +Prompting is the distinctive part of this checkpoint: it expects a structured +JSON caption rather than an unexpanded natural-language prompt, passed as the +request's `prompt` string. A compact 17-frame, 12-step smoke-test profile is +available for quick validation next to the released 40-step profile. + +The public checkpoint served by the pipeline: | Model ID | Task | Default output | | --- | --- | --- | diff --git a/docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx b/docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx index 1735640e2..c26f33c3d 100644 --- a/docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx +++ b/docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx @@ -7,13 +7,13 @@ tag: REALTIME import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + ## 1. Model Introduction -lingbot-world-v2-14b-causal-fast-diffusers is a realtime camera-controlled video world model. In SGLang-diffusion, it belongs to the realtime causal path: the server keeps a live session, samples control signals per chunk, reuses causal DiT state, and decodes video frames incrementally. +[LingBot World 2.0](https://huggingface.co/robbyant/lingbot-world-v2-14b-causal-fast-diffusers) is the 14B realtime camera-controlled world model in the LingBot family. It generates continuously from a live causal session, accepts control updates between chunks, and reuses DiT and VAE state for incremental frame delivery. -This is different from offline diffusion video models such as Wan or LTX. Offline models denoise a bounded latent sequence for one request. Realtime world models generate a continuing stream, so the runtime must manage session state, control events, causal attention cache, and VAE decode cache. +Choose 2.0 when the larger model and interactive KV-window path justify an 8-GPU deployment. Like the earlier LingBot release, it prioritizes responsive control and bounded causal state over the full-clip global context of offline video diffusion. ## 2. Deployment diff --git a/docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx b/docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx index 6e3313d1d..1f9cbb83b 100644 --- a/docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx +++ b/docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx @@ -7,13 +7,13 @@ tag: REALTIME import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + ## 1. Model Introduction -[LingBot World](https://huggingface.co/robbyant/lingbot-world-fast-diffusers) is a realtime camera-controlled video world model. In SGLang-diffusion, it belongs to the realtime causal path: the server keeps a live session, samples control signals per chunk, reuses causal DiT state, and decodes video frames incrementally. +[LingBot World](https://huggingface.co/robbyant/lingbot-world-fast-diffusers) is a realtime camera-controlled video world model. It keeps a live causal session, applies prompt and camera events between chunks, reuses DiT/VAE state, and streams decoded frames instead of finishing a bounded clip before returning. -This is different from offline diffusion video models such as Wan or LTX. Offline models denoise a bounded latent sequence for one request. Realtime world models generate a continuing stream, so the runtime must manage session state, control events, causal attention cache, and VAE decode cache. +Choose it for interactive exploration and continuous control, not one-shot cinematic generation. The causal window makes long sessions practical but gives up the global bidirectional context available to offline Wan or LTX pipelines; session state and cache policy are therefore part of the serving contract. ## 2. Deployment diff --git a/docs/cookbook/diffusion/LongLive/LongLive-2.0.mdx b/docs/cookbook/diffusion/LongLive/LongLive-2.0.mdx index fdb02c305..c1bdfc424 100644 --- a/docs/cookbook/diffusion/LongLive/LongLive-2.0.mdx +++ b/docs/cookbook/diffusion/LongLive/LongLive-2.0.mdx @@ -5,13 +5,15 @@ description: "Serve LongLive 2.0 distilled text-to-video and image-to-video mode import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + ## 1. Model Introduction -[LongLive 2.0](https://nvlabs.github.io/LongLive/LongLive2/) is a distilled few-step text-to-video and image-to-video model from NVIDIA, built on Wan2.2-TI2V-5B. SGLang serves the Diffusers-format conversion for single-prompt and multi-shot video generation. +[LongLive 2.0](https://nvlabs.github.io/LongLive/LongLive2/) is NVIDIA's 4-step text/image-to-video model distilled from Wan2.2-TI2V-5B. Its main strength is extending few-step causal generation across prompt changes, so a single request can produce multi-shot sequences without paying a full diffusion schedule for every shot. -For more details, check the [LongLive 2.0 paper](https://arxiv.org/abs/2605.18739) and [LongLive 2.0 GitHub](https://github.com/NVlabs/LongLive). The model weights are released under the NVIDIA Open Model License. +Choose it for low-step long or multi-shot generation rather than maximum one-shot fidelity. The SGLang path uses a Diffusers conversion of the official weights, and scene continuity still depends on prompt-block and sink settings; validate transitions on the target storyboard. + +The model weights use the NVIDIA Open Model License. See the [paper](https://arxiv.org/abs/2605.18739) and [GitHub repository](https://github.com/NVlabs/LongLive) for training details. ## 2. SGLang-diffusion Installation diff --git a/docs/cookbook/diffusion/MOVA/MOVA.mdx b/docs/cookbook/diffusion/MOVA/MOVA.mdx index 355dd7c9b..fb0053ac9 100644 --- a/docs/cookbook/diffusion/MOVA/MOVA.mdx +++ b/docs/cookbook/diffusion/MOVA/MOVA.mdx @@ -6,22 +6,18 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + ## 1. Model Introduction -[MOVA](https://github.com/OpenMOSS/MOVA) (MOSS Video and Audio) is a foundation model developed by the SII-OpenMOSS Team, designed to break the "silent era" of open-source video generation. Unlike cascaded pipelines that generate sound as an afterthought, MOVA synthesizes video and audio simultaneously in a single inference pass for perfect alignment. It adopts an Asymmetric Dual-Tower Architecture, fusing pre-trained video and audio towers through a bidirectional cross-attention mechanism to maintain tight synchronization between video and audio during generation. +[MOVA](https://github.com/OpenMOSS/MOVA) generates video and audio together with an asymmetric dual-tower model connected by bidirectional cross-attention. Its strongest use cases are speaking subjects, visible sound-producing events, and scenes where ambient audio must track the picture rather than be synthesized by a later cascade. -[MOVA-360p](https://huggingface.co/OpenMOSS-Team/MOVA-360p) is suitable for fast inference and resource-constrained environments. [MOVA-720p](https://huggingface.co/OpenMOSS-Team/MOVA-720p) provides higher resolution video generation. Both versions support generating up to 8 seconds of video-audio content. +The public 360p and 720p checkpoints both generate up to 8 seconds. Choose 360p for the lighter deployment and 720p for output resolution; MOVA is less suitable when the task needs long-form continuity or the richer image/video/audio reference conditioning provided by H3. -**Key Features:** - -- **Native Bimodal Generation**: Generates high-fidelity video and synchronized audio in a single inference pass, eliminating error accumulation from cascaded pipelines -- **Precise Lip-Sync**: Achieves state-of-the-art performance in multilingual lip-synchronization (LSE-D: 7.094, LSE-C: 7.452 with Dual CFG on Verse-Bench Set3) -- **Environment-Aware Sound Effects**: Generates corresponding environmental sound effects including physical interaction sounds, ambient sounds, and spatial/textural sound feedback -- **Fully Open-Source**: Model weights, inference code, training pipelines, and LoRA fine-tuning scripts are all open-sourced - -For more details, please refer to the [MOVA-360p HuggingFace page](https://huggingface.co/OpenMOSS-Team/MOVA-360p), the [MOVA-720p HuggingFace page](https://huggingface.co/OpenMOSS-Team/MOVA-720p), the [GitHub repository](https://github.com/OpenMOSS/MOVA), and the [technical report (arXiv)](https://arxiv.org/abs/2602.08794). +| Checkpoint | Best fit | Output limit | +| --- | --- | --- | +| `OpenMOSS-Team/MOVA-360p` | Faster, lower-memory joint audiovisual generation | Up to 8 seconds at 360p | +| `OpenMOSS-Team/MOVA-720p` | Higher-resolution lip-sync and environment audio | Up to 8 seconds at 720p | ## 2. SGLang-diffusion Installation diff --git a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx index 159fe7536..b9d3230e6 100644 --- a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx +++ b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx @@ -5,9 +5,43 @@ metatags: description: "Serve MiniMax-H3 with SGLang Diffusion for text-to-video-and-audio, first/last-frame conditioning, video-to-video, and multimodal reference conditioning." --- -## 1. Model introduction +import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/MiniMaxAI/minimax-h3.jsx"; -[MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) generates a video and a synchronized stereo audio track in one request. SGLang Diffusion provides a native pipeline for the three public task profiles, split across the released FL2VA (First-and-Last-Frame-to-Video-and-Audio) and Ref2VA (Reference-to-Video-and-Audio) checkpoint partitions: + + +## 1. Quick start + +Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose +a verified recipe below. Setup changes the deployment; Server and Request expose +orthogonal startup and sampling choices. + + + + +The generated Server command already includes the recommended encoder policy. +Change a Server option only for a deliberate trade-off; Request options do not +reload the model. + + +The Docker form installs the platform-specific diffusion extra from the source +bundled in the image. For conditioned requests, set **Host media directory** +under **Variables**; the builder mounts it read-only at `/data/minimax-h3`. +AMD currently offers the Python form, while NVIDIA also offers Docker. + +To use ModelScope through the same normal `sglang serve` path, prefix the copied +command with `SGLANG_USE_MODELSCOPE=true` and replace the model path with +`MiniMax/MiniMax-H3`. Keep the selected variant and topology flags unchanged. + +For platform-specific installation details, see the +[SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). + +## 2. Model capabilities + +[MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) is a native joint video-and-audio model for text-to-video-and-audio, first/last-frame control, and multimodal reference conditioning. Its main strength is producing the picture and stereo soundtrack together, so speech, music, ambient sound, and visible events can stay aligned without a separate audio-generation pass. + +Choose H3 when synchronized audiovisual output or reference-driven generation matters more than a lightweight deployment. The released recipe targets a 768-pixel short edge at 24 fps for 4–15 seconds, and its capabilities are split across two checkpoint partitions; serving every mode therefore requires separate FL2VA and Ref2VA deployments. | Task | `task` value | Conditioning | | --- | --- | --- | @@ -29,64 +63,12 @@ mapping; do not point `--model-path` at a manually downloaded subdirectory. Review the license and usage terms in the MiniMax-H3 model card before production or commercial use. SGLang support does not grant additional model usage rights. -## 2. Installation +## 3. Deployment details -Install SGLang with the diffusion dependencies: - -```bash Command -uv pip install "sglang[diffusion]" --prerelease=allow -``` - -For platform-specific setup, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). - -## 3. Serve MiniMax-H3 - -Use the interactive selector to choose a hardware platform, deployment profile, -one of the two checkpoint partitions, a request mode, and deployment features. -It generates Python and, where available, Docker launch forms. AMD selections -use the Python form until an H3-capable ROCm image is validated. The **$ cURL** -button follows the selected request mode and switches the payload across -text-only, all three first/last-frame signatures, and the image/audio/video -reference combinations listed below. -Set **Outputs per prompt** in the picker’s **Env** panel to generate more than -one output without mixing request sampling controls into the deployment -matrix. - -The Docker form does not assume the base SGLang image contains optional -diffusion dependencies. It installs the platform-specific diffusion extra from -the source bundled in the image before starting the server. Set **Host media -directory** in the **Env** panel for FL2VA, V2V, or Ref2VA; the picker mounts -that directory read-only at `/data/minimax-h3` inside the container. - -Every hardware/topology cell in this picker has completed a real request on -that exact GPU model. Approximate load-time features such as online -quantization are called out separately in the generated command. Sampling -behavior such as Cache-DiT is documented separately below. - -**Deployment Profile** exposes resident and FSDP placement on B200, B300, -H200, and H100. Resident is the latency-oriented default; FSDP reduces DiT -weight residency at the cost of per-block parameter collectives. On H200 it -also selects the verified 2-node cross-node topology. **Online -Quantization** appears only on B200 and B300. AMD keeps its resident AITER -recipe, while RTX 5090 uses its dedicated layerwise-offload profile. A -single 24 GB card (RTX 4090) uses the same offload knobs plus online -`kitchen_int8`; that recipe is documented below rather than in the picker. - -import { Deployment } from "/src/snippets/_deployment.jsx"; -import { config } from "/src/snippets/configs/MiniMaxAI/minimax-h3.jsx"; - - - - -The ready-to-run request template lives behind the **$ cURL** button in the -picker above. It regenerates as you change the selection, so the payload it -shows always matches the serve command next to it. - - -The selector uses the verified Hugging Face ID. To use ModelScope through the -same normal `sglang serve` path, prefix the copied command with -`SGLANG_USE_MODELSCOPE=true` and replace the model path with -`MiniMax/MiniMax-H3`; keep its selected variant and topology flags unchanged. +The builder accepts legal custom GPU counts and topologies, marking them +**Unverified** until the exact recipe has completed end-to-end validation. +Static H3 head or partition violations disable Copy before they reach +`sglang serve`. For a four-card H200 host, keep the full BF16/FP32 model resident by default. The model fits without FSDP, so this path avoids the per-block parameter @@ -98,6 +80,7 @@ sglang serve \ --model-variant fl2va \ --num-gpus 4 \ --ulysses-degree 4 \ + --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` @@ -121,6 +104,7 @@ sglang serve \ --num-gpus 4 \ --tp-size 2 \ --ulysses-degree 2 \ + --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` @@ -136,6 +120,7 @@ sglang serve \ --model-variant fl2va \ --num-gpus 4 \ --ulysses-degree 4 \ + --encoder-parallel auto \ --performance-mode speed \ --use-fsdp-inference true \ --port 30010 @@ -153,6 +138,7 @@ sglang serve \ --num-gpus 2 \ --tp-size 2 \ --ulysses-degree 1 \ + --encoder-parallel auto \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder,vae \ --dit-offload-prefetch-size 1 \ @@ -680,6 +666,7 @@ sglang serve \ --sp-degree 4 \ --ulysses-degree 4 \ --ring-degree 1 \ + --encoder-parallel auto \ --performance-mode speed \ --use-fsdp-inference false \ --enable-torch-compile false \ @@ -764,6 +751,7 @@ sglang serve \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ + --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` @@ -777,7 +765,11 @@ task profile. A real B200 request has completed, but the `quality: "high"` path above remains fail-closed to the audited 4×H200 workload. -## 7. Runtime feature recipes +## 7. Feature contracts and advanced recipes + +The generated command already contains the recommended topology and encoder +setting. Use the detailed reference below only when applying an optional +override or checking its installation, topology limits, and validation evidence. @@ -791,7 +783,6 @@ listed hardware and topology; it is not inherited by a similar GPU family. | --- | --- | --- | | Ulysses sequence parallelism | Verified: 8× B200, 4× H200, 4× H100, and Ulysses1/2/4/8 on MI300X and MI355X | Use `--ulysses-degree`. Combine with Ring for cross-node scaling; see the next row. | | Ring sequence parallelism (cross-node) | Verified: 2 nodes of 8× H200 each (Ulysses8 × Ring2) | Use `--ring-degree` together with `--nnodes`/`--node-rank`/`--dist-init-addr`. Ring shards the sequence across nodes while Ulysses shards heads within a node; H3's packed multi-segment attention only supports Ring across the node boundary, not within a single node's Ulysses group. Requires `--encoder-parallel replicate` — `auto`'s fold decision is not node-boundary aware. See the benchmark section below. | -| SageAttention | Supported | Use `--attention-backend sage_attn` to select the native packed varlen path; install the SageAttention dependency first. | | Tensor parallelism | Verified: B200 TP2 + Ulysses4; H100 TP2 + Ulysses2 and TP4 + Ulysses1 | `--tp-size` may be combined with Ulysses when the TP-local head count remains divisible by the Ulysses degree. On 4×H100, TP2 + Ulysses2 is the measured speed default. | | FSDP inference | Verified: 4× B200 and 4× H100 + Ulysses4 | Preserves H3's mixed BF16/FP32 parameter policy. B200 completed the exact eager comparison; H100 completed consecutive real requests at about 57 GB peak memory per GPU. | | Resident components | Verified: B200, H200, 4×H100 with TP, and 1/2/4/8× MI300X and MI355X | This is the recommended single-request latency path when the complete workload fits. | @@ -814,6 +805,7 @@ sglang serve \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ + --encoder-parallel auto \ --performance-mode speed \ --enable-breakable-cuda-graph true \ --warmup-resolutions 1344x768 \ @@ -827,6 +819,29 @@ different task profile, reference set, resolution, or prompt template. + + +Leave `--attention-backend` unset for the platform default. Use +`--attention-backend fa` only for an explicit FlashAttention comparison. + +SageAttention uses quantized attention math and is not a consistency mode. To +select H3's native packed-varlen Sage path, install the dependency and add +`--attention-backend sage_attn`. On Hopper, install the upstream SM90 binding +fix rather than the PyPI 2.2.0 build: + +```bash Command +pip install --force-reinstall \ + git+https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4 \ + --no-build-isolation +``` + +The backend is a server-wide default. Use +`--component-attention-backends` only when a measured component needs a +different kernel, and keep the platform default for every component not named +in the override. + + + On the verified 8× B200 topology, quantize the BF16 transformer at server load: @@ -837,6 +852,7 @@ sglang serve \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ + --encoder-parallel auto \ --performance-mode speed \ --quantization fp8 \ --port 30010 @@ -852,6 +868,7 @@ sglang serve \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ + --encoder-parallel auto \ --quantization fp8 \ --quantization-ignored-layers blocks.0.attn token_refiner \ --port 30010 @@ -861,8 +878,8 @@ sglang serve \ Online FP8 is approximate and is not a consistency ground-truth mode. It can be combined with Cache-DiT, but the two approximations compound. Validate visual quality, audio quality, memory use, and latency on the target workload. -The picker exposes this option only on the B200 and B300 topologies used for -real H3 validation runs. +This recipe is limited to the resident B200 and B300 topologies used for real +H3 validation runs. On a single 24 GB card, use `kitchen_int8` instead of FP8. It quantizes the @@ -924,6 +941,33 @@ the strict `quality="high"` deployment contract. + + +The picker explicitly writes `--encoder-parallel auto` in every single-node +recipe. At the default request batch size of one, H100/H200/B200/B300 servers +with peer-to-peer access fold the Qwen encoder across otherwise idle Ulysses +ranks. A pure-TP recipe keeps the encoder inside its TP group, while a +PCIe-only host can avoid an expensive world fold. Keep `auto` unless one of +the cases below applies. + +Encoder DP is a throughput policy for compatible request batches. It requires +TP1 and DiT DP1, replicates the encoder weights, and does not improve a batch +of one: + +```bash Overlay +--encoder-parallel dp \ +--batching-max-size 2 +``` + +The cross-node picker recipe already uses replication because the automatic +fold decision is not node-boundary aware: + +```bash Overlay +--encoder-parallel replicate +``` + + + ## 8. Configuration notes @@ -995,7 +1039,7 @@ The swept axes are `--model-variant` (`fl2va` / `ref2va`), `--quantization` This is a single-request latency sweep (`batching_max_size: 1`), so encoder DP is intentionally excluded: it cannot distribute a batch of one. Use the -picker’s **DP (batched throughput)** option for a multi-request throughput +**DP for a request batch** setting above for a compatible multi-request deployment; the table below does not claim a measured H3 DP speedup. **Driver.** diff --git a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx index f25dbebd6..50d0d82b5 100644 --- a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx +++ b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx @@ -7,22 +7,13 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; import { QwenImageEditDeployment } from '/src/snippets/diffusion/qwen-image-edit-deployment.jsx'; - + ## 1. Model Introduction -[Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) is an enhanced version over Qwen-Image-Edit-2509, featuring multiple improvements—including notably better consistency. Built upon the 20B Qwen-Image model, Qwen-Image-Edit-2511 successfully extends Qwen-Image's unique text rendering capabilities to image editing tasks, enabling precise text editing. +[Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) is the 20B editing counterpart to Qwen-Image. It is strongest at changing text, materials, lighting, viewpoint, or composition while reducing drift in regions that were not requested to change. -Key Enhancements in Qwen-Image-Edit-2511: - -- **Mitigate Image Drift**: Reduces unwanted changes in non-edited regions of the image. -- **Improved Character Consistency**: The model can perform imaginative edits based on an input portrait while preserving the identity and visual characteristics of the subject. -- **Multi-Person Consistency**: Enhanced consistency in multi-person group photos, enabling high-fidelity fusion of two separate person images into a coherent group shot. -- **Integrated LoRA Capabilities**: Selected popular community-created LoRAs are integrated directly into the base model, unlocking their effects without extra tuning (e.g., lighting enhancement, viewpoint generation). -- **Enhanced Industrial Design Generation**: Special attention to practical engineering scenarios, including batch industrial product design and material replacement for industrial components. -- **Strengthened Geometric Reasoning**: Stronger geometric reasoning capability for generating auxiliary construction lines for design or annotation purposes. - -For more details, please refer to the [official Qwen-Image-Edit-2511 HuggingFace page](https://huggingface.co/Qwen/Qwen-Image-Edit-2511), the [Blog](https://qwenlm.github.io/blog/qwen-image-edit-2511/), and the [Tech Report](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/Qwen_Image.pdf). +Choose it for identity-sensitive portrait edits, multi-person composition, typography replacement, and geometry-aware design work. It is substantially heavier than small specialist editors, and consistency is improved rather than guaranteed; evaluate untouched-region drift on the actual editing workload. ## 2. SGLang-diffusion Installation diff --git a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx index ba657ed11..4703ca5e1 100644 --- a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx +++ b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx @@ -7,13 +7,13 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; import { QwenImageDeployment } from '/src/snippets/diffusion/qwen-image-deployment.jsx'; - + ## 1. Model Introduction -[Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) is a text-to-image diffusion model developed by the Qwen team. +[Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) is a 20B text-to-image model built for strong prompt following and precise rendering of English and Chinese text. It is especially useful for posters, signs, diagrams, and dense layouts where typography and spatial relationships matter as much as general image quality. -For more details, please refer to the [official Qwen-Image HuggingFace page](https://huggingface.co/Qwen/Qwen-Image), the [Blog](https://qwenlm.github.io/blog/qwen-image/), and the [Tech Report](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/Qwen_Image.pdf). +This page covers generation rather than editing: use Qwen-Image-Edit when an existing image, subject identity, or untouched region must be preserved. The full checkpoint is memory-heavy, while the validated ModelOpt NVFP4 release provides a supported low-precision deployment option with an expected quality tradeoff. ## 2. SGLang-diffusion Installation diff --git a/docs/cookbook/diffusion/README.mdx b/docs/cookbook/diffusion/README.mdx index 3ef955fbc..c26aaa21c 100644 --- a/docs/cookbook/diffusion/README.mdx +++ b/docs/cookbook/diffusion/README.mdx @@ -1,36 +1,44 @@ --- title: "Diffusion Cookbook" -description: "Cookbook recipes for running diffusion models with SGLang" +description: "Author and review deployment-focused diffusion model recipes for SGLang." metatags: - description: "Explore SGLang diffusion cookbook structure, categories, and contribution guidance for image and video generation recipes." + description: "SGLang diffusion cookbook structure, model-page standards, and contribution guidance." --- # SGLang Diffusion Cookbook -
- License - PRs Welcome -
+The diffusion cookbook is a set of model-specific, validated deployment guides. Each page +should let a reader answer three questions quickly: what the model is good at, which +checkpoint and base topology to choose, and which optional features change performance, +memory, or quality. -Create a comprehensive cookbook for diffusion models in SGLang, demonstrating SGLang's performance advantages for image and video generation workloads. +## What every model page includes -## 🎯 What You'll Find Here +1. A compact capability tag row using the shared `DiffusionModelTags` component. +2. A short introduction that states the model's capability range, strongest use case, and + an important limitation or deployment tradeoff. +3. Verified base launch recipes for the hardware actually tested. +4. Matching request examples for every public task or checkpoint partition. +5. Optional attention, quantization, caching, compile, and offload recipes with explicit + quality contracts and validation scope. -This cookbook aggregates battle-tested SGLang recipes covering: +Tags describe the model, not every runtime flag. Prefer `video + audio`, `multi-reference`, +or `4–15 seconds` over generic labels such as `native`, `fast`, or `high quality`. -- **Models**: Mainstream Image and Video generation Models -- **Use Cases**: Inference serving, deployment strategies -- **Hardware**: GPU and CPU configurations, optimization for different accelerators -- **Best Practices**: Configuration templates, performance tuning, troubleshooting guides +## Keep the command picker small -Each recipe provides step-by-step instructions to help you quickly implement SGLang solutions for your specific requirements. +The command picker selects a runnable base recipe: hardware topology, checkpoint partition, +request mode, and a validated placement profile. Independent feature knobs do not need to +become picker dimensions. -## 🚀 Quick Start +Document orthogonal controls as feature overlays instead: -1. Browse the recipe index above to find your model -2. Follow the step-by-step instructions in each guide -3. Adapt configurations to your specific hardware and requirements -4. Join our community to share feedback and improvements +| Feature class | Where it belongs | Required context | +| --- | --- | --- | +| Attention backend | Optional feature overlays | Default backend, precision/approximation contract, installation, measured hardware | +| Online or pre-quantized weights | Optional feature overlays | Weight source, protected precision-sensitive layers, quality and memory tradeoff | +| Cache-DiT and sampling controls | Request/sampling recipes | Quality level, incompatible placement modes, task-specific validation | +| TP, Ulysses, Ring, FSDP, residency | Base picker only for verified cells | Exact GPU count, topology, memory, and latency evidence | For memory placement, prefer the unified [`--component-residency`](/docs/sglang-diffusion/api/cli#component-residency) @@ -40,62 +48,25 @@ selector. Each component resolves to exactly one of `resident`, `--image-encoder-cpu-offload`, and `--vae-cpu-offload` remain supported by all recipes that already use them. -The sglang diffusion cookbook directory structure are shown below: +## Contributing a model -```text Example -docs/cookbook/diffusion/ -├── README.mdx # Main cookbook (this file) -├── Qwen-Image/ # Qwen-Image series docs -│ ├── Qwen-Image.mdx -│ └── Qwen-Image-Edit.mdx -├── Wan/ # Wan series docs -│ ├── Wan2.1.mdx -│ └── Wan2.2.mdx -├── Z-Image/ # Z-Image series docs -│ └── Z-Image-Turbo.mdx -├── Ernie-Image/ # ERNIE-Image series docs -│ └── Ernie-Image.mdx -└── ... +Use the repo-local `cookbook-add-model` skill and its +`templates/diffusion-page.mdx.tmpl` template. The full category contract lives in +`.claude/skills/cookbook-add-model/references/diffusion-authoring.md`. + +Before opening a PR, run: + +```bash Command +node docs/scripts/check_cookbook_configs.mjs +cd docs +mint validate +mint broken-links --check-anchors --check-redirects ``` -## 🤝 Contributing +The static cookbook check requires every diffusion model page to use the shared tag widget, +the standard introduction heading, 4–6 tags, and a substantive two-paragraph lead. Mintlify +validation then catches MDX, navigation, and internal-link problems. -We believe the best documentation comes from practitioners. Whether you've optimized SGLang for a specific model, solved a tricky deployment challenge, or discovered performance improvements, we encourage you to contribute your recipes! - -**💪How to Contribute** - -- Comment below if interested (mention which role) -- Join discussion on implementation details -- Fork repo and work on assigned section -- Submit PR following SGLang cookbook standards -- Iterate based on review feedback - -**To contribute:** - -```shell Command -# Fork the repo and clone locally -git clone https://github.com/YOUR_USERNAME/sglang.git -cd sglang - -# Create a new branch -git checkout -b add-my-recipe - -# Add your recipe under docs/cookbook/diffusion/ -# Submit a PR! -``` - -## 📖 Resources - -- [SGLang GitHub](https://github.com/sgl-project/sglang) -- [SGLang Documentation](/) -- [SGLang Diffusion Documentation](/docs/sglang-diffusion/index) -- [SLACK Channel](https://sgl-fru7574.slack.com/archives/C07GLLLESNR) -- [Community Slack/Discord](https://discord.gg/MpEEuAeb) - -## 📄 License - -This project is licensed under the Apache License 2.0 - see the [LICENSE](https://github.com/sgl-project/sglang/blob/main/LICENSE) file for details. - ---- - -**Let's build this resource together!** 🚀 Star the repo and contribute your recipes to help the SGLang community grow. +For runtime concepts and platform support, see the +[SGLang Diffusion documentation](/docs/sglang-diffusion/index) and +[compatibility matrix](/docs/sglang-diffusion/compatibility_matrix). diff --git a/docs/cookbook/diffusion/SANA-Video/SANA-Video.mdx b/docs/cookbook/diffusion/SANA-Video/SANA-Video.mdx index ad3c67ab6..471b59ddb 100644 --- a/docs/cookbook/diffusion/SANA-Video/SANA-Video.mdx +++ b/docs/cookbook/diffusion/SANA-Video/SANA-Video.mdx @@ -7,12 +7,20 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + -## 1. Model introduction +## 1. Model Introduction [SANA-Video 2B 480p](https://huggingface.co/Efficient-Large-Model/SANA-Video_2B_480p_diffusers) -is a text-to-video model with a native SGLang Diffusion pipeline. +is a lightweight text-to-video model served through a native SGLang Diffusion +pipeline. Its main advantage is deployment cost: at 2B parameters it serves +480p video from a single GPU with a plain `sglang serve` invocation, no +parallelism flags required. + +The released generation profile produces 832×480 output at 81 frames and +16 FPS over 50 inference steps; a compact 17-frame, 8-step profile is covered +by server CI for quick validation. Motion strength can be steered directly +from the prompt with an optional `motion score: N.` suffix. | Model ID | Task | Default output | | --- | --- | --- | diff --git a/docs/cookbook/diffusion/SANA-WM/SANA-WM.mdx b/docs/cookbook/diffusion/SANA-WM/SANA-WM.mdx index 959899b78..970894f4f 100644 --- a/docs/cookbook/diffusion/SANA-WM/SANA-WM.mdx +++ b/docs/cookbook/diffusion/SANA-WM/SANA-WM.mdx @@ -6,32 +6,19 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; - + ## 1. Model Introduction -[SANA-WM](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) is an efficient open-source **world model** from NVLabs, trained natively for one-minute video generation. It is a **2.6B-parameter text+image-to-video (TI2V) diffusion transformer** that synthesizes **720p, minute-scale videos with precise 6-DoF camera control**, paired with an **LTX-2 refiner** for high-fidelity decoding. It builds on the [SANA](https://github.com/NVlabs/Sana) family — efficient high-resolution synthesis with a linear diffusion transformer. +[SANA-WM](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) is NVLabs' 2.6B text-and-image-to-video world model for 720p, minute-scale generation with explicit per-frame 6-DoF camera control. Its hybrid recurrent/softmax attention keeps long causal histories bounded, while an LTX-2 refiner supplies the detail that the fast Stage-1 world model does not produce on its own. -SANA-WM ships in two checkpoints: a **bidirectional** checkpoint (dense, one-shot) and a **streaming** checkpoint (chunk-causal, autoregressive — generated chunk-by-chunk, reusing causal DiT state across chunks for bounded memory → long, even endless, clips). From a single first frame, a text prompt, and a camera trajectory, this cookbook covers **all three serving modes** SGLang exposes: +Choose the dense checkpoint for the best bounded-clip quality and the streaming checkpoint for long-running or interactive control. Streaming and realtime trade global bidirectional context for bounded state and lower response latency; the realtime WebSocket path is not bit-identical to offline batch streaming. -- **(A) Dense bidirectional** (§4) — the `SANA-WM_bidirectional` checkpoint generated in one shot (no chunking) via **`SanaWMTwoStagePipeline`** over the standard **`/v1/videos`** HTTP API. Highest single-clip quality (full bidirectional attention + dense LTX-2 refiner); matches the NVlabs dense reference. -- **(B) Batch streaming** (§5) — the `SANA-WM_streaming` checkpoint generated chunk-by-chunk in one request via the same **`SanaWMTwoStagePipeline`** + `--streaming` over **`/v1/videos`**. This is SGLang's offline chunk-causal streaming path: the whole clip is produced chunk-by-chunk internally, then returned. -- **(C) Live realtime** (§6–7) — the streaming pipeline exposed as **`SanaWMRealtimePipeline`** over a **WebSocket API** at `/v1/realtime_video/generate`, so a browser/client streams camera-action events frame-by-frame and receives video chunks back in real time. Realtime uses the same streaming checkpoint, but the incremental session path is not bit-identical to offline batch streaming. - -All three modes share the camera action DSL (§8) and the configuration knobs (§9). Modes (B) and (C) share the streaming checkpoint and the chunk-causal pipeline. - -**Key features** (per the official model): - -- **Hybrid Linear Attention** — frame-wise Gated DeltaNet (GDN) recurrent blocks combined with softmax attention (every 4th layer, block indices {3,7,11,15,19}) for memory-efficient long-context modeling. -- **Dual-Branch Camera Control** — independent main and camera branches (UCPE + PRoPE) for precise per-frame 6-DoF trajectory adherence. -- **Two-Stage Pipeline** — an LTX-2 long-video refiner on top of Stage-1 latents for quality and temporal consistency. - -In the **streaming / realtime** configuration this becomes a low-latency, interactive pipeline: - -- **Stage-1 chunk-causal DiT** — the streaming path carries a **per-block KV cache** (recurrent GDN state + a softmax K/V window) across chunks; bounded memory means it scales to long / endless sequences. Stage-1 is intentionally coarse. -- **LTX-2 streaming refiner** — refines each Stage-1 latent chunk block-by-block with a **sink + sliding-history KV cache** (required for sharp output). -- **Causal LTX-2 VAE** — decodes latents chunk-by-chunk with a carried conv-cache for seam-free frames. -- **Camera control** — drive the camera with a compact **WASD/IJKL** action DSL (move with WASD, look with IJKL; see §8) — supplied at request time on the `/v1/videos` paths, or pushed over the WebSocket at init / as live per-chunk events on the realtime path (see §7). +| Serving mode | Checkpoint and API | Best fit | Tradeoff | +| --- | --- | --- | --- | +| Dense bidirectional | `SANA-WM_bidirectional` via `/v1/videos` | Highest single-clip quality | Bounded one-shot generation; full attention cost | +| Batch streaming | `SANA-WM_streaming` via `/v1/videos` | Long clips returned as one job | Chunk-causal context rather than global bidirectional context | +| Live realtime | `SANA-WM_streaming` via WebSocket | Interactive camera events and incremental output | Stateful session; not bit-identical to batch streaming | **Architecture & components** @@ -44,7 +31,7 @@ In the **streaming / realtime** configuration this becomes a low-latency, intera | Refiner | LTX-2 Stage-2 distilled; ~41 GB | | Output | up to 720p (704×1280) @ 16 fps, minute-scale | -For more details, see the [SANA-WM paper (arXiv)](https://arxiv.org/abs/2605.15178), the [SANA project page](https://nvlabs.github.io/Sana/), the [NVlabs/Sana GitHub](https://github.com/NVlabs/Sana), and the [SANA-WM_bidirectional model card](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) (Apache-2.0). +For architecture and training details, see the [SANA-WM paper](https://arxiv.org/abs/2605.15178) and [model card](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) (Apache-2.0). ## 2. Installation diff --git a/docs/cookbook/diffusion/Wan/Wan2.1.mdx b/docs/cookbook/diffusion/Wan/Wan2.1.mdx index de9b21aef..bd1e85726 100644 --- a/docs/cookbook/diffusion/Wan/Wan2.1.mdx +++ b/docs/cookbook/diffusion/Wan/Wan2.1.mdx @@ -7,24 +7,13 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; import { Wan21Deployment } from '/src/snippets/diffusion/wan21-deployment.jsx'; - + ## 1. Model Introduction -[Wan2.1 series](https://github.com/Wan-Video/Wan2.1) is an open and advanced suite of large-scale video generative models from Wan-AI. +[Wan2.1](https://github.com/Wan-Video/Wan2.1) is a broad open video family covering text-to-video and image-to-video across 1.3B and 14B checkpoints. Its practical strengths are motion-rich generation, temporal consistency, and readable Chinese/English text, with 480p and 720p variants for different quality and memory targets. -Key characteristics: - -- **State-of-the-art video quality**: Consistently outperforms many open-source and commercial video models on internal and public benchmarks, especially for motion richness and temporal consistency. -- **Consumer GPU friendly**: The T2V-1.3B variant can generate 5-second 480P videos on consumer GPUs with modest VRAM requirements. -- **Multi-capability suite**: Supports Text-to-Video (T2V), Image-to-Video (I2V), video editing, text-to-image, and video-to-audio generation. -- **Robust text rendering**: First-generation Wan model capable of generating both Chinese and English text in videos with strong readability. -- **Powerful Wan-VAE**: A 3D causal VAE that encodes/decodes long 1080P videos while preserving temporal information, enabling efficient high-resolution video generation. - -For more details, refer to the official Wan2.1 resources: - -- **GitHub**: [Wan-Video/Wan2.1](https://github.com/Wan-Video/Wan2.1) -- **Hugging Face collection**: [Wan-AI Wan2.1](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) +Choose the 1.3B T2V model for consumer-GPU experiments and the 14B models when quality matters more than footprint. Wan2.1 is a dense DiT family; for timestep-specialized MoE capacity or the unified 5B TI2V path, use Wan2.2 instead. ## 2. SGLang-diffusion Installation diff --git a/docs/cookbook/diffusion/Wan/Wan2.2.mdx b/docs/cookbook/diffusion/Wan/Wan2.2.mdx index 6e666869a..b3dddb0d1 100644 --- a/docs/cookbook/diffusion/Wan/Wan2.2.mdx +++ b/docs/cookbook/diffusion/Wan/Wan2.2.mdx @@ -7,20 +7,13 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; import { Wan22Deployment } from '/src/snippets/diffusion/wan22-deployment.jsx'; - + ## 1. Model Introduction -[Wan2.2 series](https://github.com/Wan-Video/Wan2.2) are the most popular and open and advanced large-scale video generative models. +[Wan2.2](https://github.com/Wan-Video/Wan2.2) extends the Wan family with timestep-specialized Mixture-of-Experts models and a unified 5B TI2V checkpoint. It targets cinematic composition, lighting, and complex motion while supporting text-to-video, image-to-video, and hybrid text/image conditioning. -This generation delivers comprehensive upgrades across the board: - -- **Effective MoE Architecture**: Introduces a Mixture-of-Experts (MoE) architecture into video diffusion models. By separating the denoising process cross timesteps with specialized powerful expert models, this enlarges the overall model capacity while maintaining the same computational cost. -- **Cinematic-level Aesthetics**: Incorporates meticulously curated aesthetic data, complete with detailed labels for lighting, composition, contrast, color tone, and more. This allows for more precise and controllable cinematic style generation, facilitating the creation of videos with customizable aesthetic preferences. -- **Complex Motion Generation**: Trained on a significantly larger data, with +65.6% more images and +83.2% more videos. This expansion notably enhances the model's generalization across multiple dimensions such as motions, semantics, and aesthetics, achieving TOP performance among all open-sourced and closed-sourced models. -- **Efficient High-Definition Hybrid TI2V**: Open-sources a 5B model built with our advanced Wan2.2-VAE that achieves a compression ratio of 16×16×4. This model supports both text-to-video and image-to-video generation at 720P resolution with 24fps and can also run on consumer-grade graphics cards like 4090. It is one of the fastest 720P@24fps models currently available, capable of serving both the industrial and academic sectors simultaneously. - -For more details, please refer to the [official Wan2.2 GitHub Repository](https://github.com/Wan-Video/Wan2.2). +Choose the A14B MoE checkpoints for maximum T2V or I2V capacity and the 5B TI2V model for a smaller unified 720p-at-24-fps path. MoE reduces active compute relative to total capacity but does not remove the memory cost of loading expert weights, so hardware selection still matters. ## 2. SGLang-diffusion Installation diff --git a/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx b/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx index 255406a0e..7196cda2d 100644 --- a/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx +++ b/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx @@ -7,23 +7,13 @@ metatags: import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; import { ZImageTurboDeployment } from '/src/snippets/diffusion/zimage-turbo-deployment.jsx'; - + ## 1. Model Introduction -[Z-Image](https://github.com/Tongyi-MAI/Z-Image) is a powerful and highly efficient image generation model family with 6B parameters, developed by Tongyi-MAI. It adopts a Scalable Single-Stream DiT (S3-DiT) architecture, where text, visual semantic tokens, and image VAE tokens are concatenated at the sequence level to serve as a unified input stream, maximizing parameter efficiency compared to dual-stream approaches. +[Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo) is a distilled 6B single-stream DiT for fast text-to-image generation. It reaches its intended operating point in 8 function evaluations and is particularly strong at photorealistic scenes, prompt adherence, and English/Chinese text rendering. -[Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo) is a distilled version of Z-Image that matches or exceeds leading competitors with only 8 NFEs (Number of Function Evaluations). It is powered by two core techniques: **Decoupled-DMD** (few-step distillation) and **DMDR** (fusing DMD with Reinforcement Learning). - -**Key Features:** - -- **Sub-second Inference Latency**: Achieves sub-second inference on enterprise-grade H800 GPUs and fits comfortably within 16GB VRAM consumer devices -- **Photorealistic Image Generation**: Excels in high-quality photorealistic image generation with rich aesthetics -- **Bilingual Text Rendering**: Supports accurate bilingual text rendering in both English and Chinese -- **Robust Instruction Adherence**: Strong prompt following and instruction adherence capabilities -- **#1 Open-Source Model**: Ranked 8th overall and #1 among open-source models on the [Artificial Analysis Text-to-Image Leaderboard](https://artificialanalysis.ai/image/leaderboard/text-to-image) - -For more details, please refer to the [Z-Image-Turbo HuggingFace page](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo), the [GitHub repository](https://github.com/Tongyi-MAI/Z-Image), and the [technical report (arXiv)](https://arxiv.org/abs/2511.22699). +Choose it when latency and a relatively small deployment footprint matter more than the editability or maximum capacity of larger image models. It is a generation-only checkpoint; use Qwen-Image-Edit or FLUX.2 when the request includes source images or identity-preserving edits. ## 2. SGLang-diffusion Installation diff --git a/docs/custom.css b/docs/custom.css index 5c6cb71e4..97ce17aa9 100644 --- a/docs/custom.css +++ b/docs/custom.css @@ -168,15 +168,22 @@ html.dark table tbody tr:nth-child(even) td, flex-wrap: wrap; align-items: center; gap: 6px; - margin: -1.5rem 0 0; + margin: 8px 0 30px; } .sgd-model-tags .sgd-chip { margin: 0; + padding: 4px 9px; + border: 0; + background: #eef2ff; + color: #3730a3; + font-size: 12px; + font-weight: 750; + line-height: 1.4; } .sgd-model-tags + h2 { - margin-top: 0.75rem; + margin-top: 0; } .prose:has(.sgd-model-tags) { @@ -184,6 +191,656 @@ html.dark table tbody tr:nth-child(even) td, overflow-y: visible; } +/* Legacy command visualizer controls. New diffusion pages can opt into the + scoped editorial builder below without changing older configs. */ +.sg-command-visualizer:not(.sgd-command-builder) :where(button:not(:disabled), .sg-command-visualizer-choice[aria-disabled="false"], .sg-command-visualizer-tab) { + transition: transform 160ms ease-out, border-color 160ms ease-out, + background-color 160ms ease-out, color 160ms ease-out; +} + +@media (hover: hover) and (pointer: fine) { + .sg-command-visualizer:not(.sgd-command-builder) :where(button:not(:disabled), .sg-command-visualizer-choice[aria-disabled="false"], .sg-command-visualizer-tab):hover { + transform: translateY(-1px) scale(1.015); + } +} + +.sg-command-visualizer:not(.sgd-command-builder) :where(button:not(:disabled), .sg-command-visualizer-choice[aria-disabled="false"], .sg-command-visualizer-tab):active { + transform: translateY(0) scale(0.985); +} + +.sg-command-visualizer:not(.sgd-command-builder) :where(button, .sg-command-visualizer-choice, .sg-command-visualizer-tab):focus-visible { + outline: 2px solid #e2775f; + outline-offset: 2px; +} + +@media (prefers-reduced-motion: reduce) { + .sg-command-visualizer:not(.sgd-command-builder) :where(button, .sg-command-visualizer-choice, .sg-command-visualizer-tab) { + transition: none; + transform: none !important; + } +} + +/* Premium responsive diffusion command builder. The component is its own + query container because Mintlify's content column is narrower than the + browser viewport whenever either navigation rail is present. */ +.sgd-command-builder { + container: command-builder / inline-size; + --sgdb-surface: white; + --sgdb-raised: white; + --sgdb-ink: oklch(22% 0.02 258); + --sgdb-muted: oklch(50% 0.02 258); + --sgdb-line: oklch(90% 0.008 258); + --sgdb-brand: oklch(57% 0.15 35); + /* Accents stay in the brand's warm family — the materials are cool + neutrals, so identity comes from where the orange lands, not from + tinting the surfaces. Serve = brand orange, Request = amber. */ + --sgdb-serve: oklch(56% 0.13 40); + --sgdb-request: oklch(60% 0.12 62); + --sgdb-ok: oklch(52% 0.11 153); + --sgdb-warn: oklch(59% 0.12 78); + --sgdb-error: oklch(53% 0.16 25); + /* Glass material system. A recessed well is a cool translucent tint (warm + greys read muddy); the raised thumb is a glass slab: gradient fill, a + specular top edge, a hairline ring, and a soft cast. The pair is + redeclared for dark, where the glass is built from white overlays. */ + --sgdb-well: oklch(45% 0.03 258 / 0.08); + --sgdb-well-inset: inset 0 1px 2px oklch(30% 0.03 258 / 0.07); + --sgdb-thumb-bg: linear-gradient(180deg, white, oklch(98.5% 0.003 258)); + --sgdb-thumb-shadow: 0 0 0 0.5px oklch(30% 0.02 258 / 0.1), + inset 0 1px 0 rgba(255, 255, 255, 0.9), + 0 1px 1px oklch(30% 0.02 258 / 0.07), + 0 4px 12px oklch(30% 0.02 258 / 0.1); + /* Hover tone for anything that sits inside a tinted well. */ + --sgdb-pick-hover: oklch(45% 0.03 258 / 0.05); + --sgdb-code-bg: oklch(97.5% 0.004 258); + --sgdb-code-inset: oklch(30% 0.03 258 / 0.06); + /* Colours settle on a decelerating curve; things that move get a spring — + a touch of overshoot is what reads as alive rather than mechanical. */ + --sgdb-ease: cubic-bezier(0.32, 0.72, 0, 1); + --sgdb-ease-dur: 200ms; + --sgdb-spring: cubic-bezier(0.3, 1.35, 0.45, 1); + --sgdb-spring-dur: 460ms; + max-width: 100%; + margin: 24px 0 40px; + overflow: clip; + border: 1px solid var(--sgdb-line); + border-radius: 16px; + background: var(--sgdb-surface); + box-shadow: 0 1px 2px oklch(30% 0.03 258 / 0.04), + 0 12px 32px -12px oklch(30% 0.03 258 / 0.12); + color: var(--sgdb-ink); +} + +.sgd-command-builder *, .sgd-command-builder *::before, .sgd-command-builder *::after { box-sizing: border-box; } +/* Form controls inherit the page typeface instead of the browser's. Wrapped in + :where() so the rule weighs (0,1,0) and every single-class control rule below + still wins on order. Written as `.sgd-command-builder button` it weighed + (0,1,1) and, being a `font` shorthand, silently reset the font-size and + font-weight of anything selected by one class — `.sgd-builder-copy` and + `.sgd-builder-text-action` asked for 11px/680 and rendered 16px/400. */ +.sgd-command-builder :where(button, input, select) { font: inherit; } + +/* Segmented control with one glass pill that glides between scopes — the + pill is a single ::before positioned off the active button via :has(), + so the slide is pure CSS and springs into place. */ +.sgd-builder-scope-tabs { + position: relative; + display: grid; + grid-template-columns: repeat(3, minmax(0, 1fr)); + margin: 12px 12px 0; + padding: 3px; + border-radius: 11px; + background: var(--sgdb-well); + box-shadow: var(--sgdb-well-inset); +} + +.sgd-builder-scope-tabs::before { + position: absolute; + top: 3px; + bottom: 3px; + left: 3px; + width: calc((100% - 6px) / 3); + border-radius: 8px; + background: var(--sgdb-thumb-bg); + box-shadow: var(--sgdb-thumb-shadow); + content: ""; + /* Glide and settle, no overshoot: a scope switch is a frequent functional + action, so the slide itself carries the life and a bounce would be noise. */ + transition: transform 300ms var(--sgdb-ease); +} + +.sgd-builder-scope-tabs:has(> button:nth-child(2)[data-active="true"])::before { transform: translateX(100%); } +.sgd-builder-scope-tabs:has(> button:nth-child(3)[data-active="true"])::before { transform: translateX(200%); } + +.sgd-builder-scope-tabs button { + position: relative; + z-index: 1; + min-width: 0; + min-height: 34px; + padding: 0 4px; + border: 0; + background: transparent; + color: var(--sgdb-muted); + cursor: pointer; + font-size: 12px; + font-weight: 590; + white-space: nowrap; + transition: color var(--sgdb-ease-dur) var(--sgdb-ease); +} + +.sgd-builder-scope-tabs button[data-active="true"] { color: var(--sgdb-ink); } +.sgd-builder-main { display: flex; min-width: 0; flex-direction: column; } +.sgd-builder-controls, .sgd-builder-output-rail { min-width: 0; } +.sgd-builder-controls { container: builder-controls / inline-size; padding: 4px 16px 20px; } + +.sgd-builder-scope-panel { display: grid; gap: 0; } +.sgd-builder-recipe { + display: flex; + min-width: 0; + padding: 11px 0; + gap: 12px; + align-items: center; + justify-content: space-between; + border-bottom: 1px solid var(--sgdb-line); +} +.sgd-builder-recipe > div:first-child { display: grid; min-width: 0; gap: 3px; } +.sgd-builder-recipe > div:first-child span { color: var(--sgdb-muted); font-size: 10.5px; } +.sgd-builder-recipe > div:first-child strong { font-size: 12px; font-weight: 590; line-height: 1.4; } +.sgd-builder-recipe > div:last-child { display: flex; flex: none; gap: 10px; align-items: center; } +.sgd-builder-recipe > div:last-child > small { color: var(--sgdb-muted); font-size: 10px; } +.sgd-builder-section { min-width: 0; padding: 13px 0; border-bottom: 1px solid var(--sgdb-line); } +.sgd-builder-section:last-child { border-bottom: 0; } +.sgd-builder-section-heading { display: flex; gap: 8px; align-items: baseline; justify-content: space-between; margin-bottom: 8px; } +.sgd-builder-section-heading > span { font-size: 12px; font-weight: 590; } +.sgd-builder-section-heading > small { color: var(--sgdb-muted); font-size: 11px; line-height: 1.35; text-align: right; } + +/* One tinted well per group instead of a border on every option. Thirteen + outlined boxes competed with each other and with the panel frame; the well + carries the grouping so each option only has to carry its own state. */ +.sgd-builder-hardware-grid, .sgd-builder-choice-grid { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 2px; + padding: 3px; + border-radius: 11px; + background: var(--sgdb-well); + box-shadow: var(--sgdb-well-inset); +} +.sgd-builder-hardware, .sgd-builder-choice { + position: relative; + display: grid; + min-width: 0; + min-height: 38px; + padding: 7px 10px; + overflow: hidden; + /* Rows in the same grid track stretch to the tallest sibling; center the + label stack so a single-line option does not ride the top of its cell. */ + align-content: center; + border: 0; + border-radius: 8px; + background: transparent; + /* Unselected stays legible — a deselected option is still a real choice, so + it sits just below the ink, not down at the muted caption level. */ + color: color-mix(in oklch, var(--sgdb-ink) 72%, transparent); + cursor: pointer; + font-size: 12px; + line-height: 1.25; + text-align: left; + transition: background-color var(--sgdb-ease-dur) var(--sgdb-ease), + color var(--sgdb-ease-dur) var(--sgdb-ease); +} + +/* Hardware label and its VRAM share one line — the second text row bought + no clarity, only cell height. Choice subtitles (checkpoint names) keep + their own line: they are long enough to need it. */ +.sgd-builder-hardware { display: flex; min-height: 34px; flex-flow: row wrap; align-items: baseline; gap: 5px; } +.sgd-builder-hardware strong { font-size: 12.5px; font-weight: 590; letter-spacing: -0.01em; } +.sgd-builder-hardware small, .sgd-builder-choice small { margin-top: 1px; color: var(--sgdb-muted); font-size: 10.5px; } +.sgd-builder-hardware small { margin-top: 0; } + +/* The radio ring is gone — the fill is the affordance. Kept in the DOM (the + JSX renders it) but collapsed, so no markup change is needed. */ +.sgd-builder-choice-dot { display: none; } + +.sgd-builder-hardware[data-selected="true"], .sgd-builder-choice[data-selected="true"] { + background: var(--sgdb-thumb-bg); + box-shadow: var(--sgdb-thumb-shadow); + color: var(--sgdb-ink); +} + +.sgd-builder-hardware:disabled, +.sgd-builder-choice:disabled, +.sgd-builder-choice[data-blocked="true"] { cursor: not-allowed; opacity: 0.42; } + +/* Tap/click feedback for a blocked option — the reason the tooltip carries, + surfaced where touch readers can see it. */ +.sgd-builder-blocked-note { + margin: 8px 2px 0; + color: var(--sgdb-warn); + font-size: 11px; + line-height: 1.45; + animation: sgdb-note-in 220ms var(--sgdb-ease); +} +@keyframes sgdb-note-in { + from { opacity: 0; transform: translateY(-2px); } + to { opacity: 1; transform: translateY(0); } +} + +/* A soft (selectable-but-unverified) pick declares itself in place. */ +.sgd-builder-choice[data-soft="true"][data-selected="true"]::after { + content: "Not verified yet"; + color: var(--sgdb-warn); + font-size: 10px; + line-height: 1.3; + margin-top: 1px; +} +.sgd-builder-resource-grid { display: grid; gap: 8px; } +.sgd-builder-stepper-field { + display: flex; + min-height: 42px; + padding: 5px 6px 5px 12px; + align-items: center; + justify-content: space-between; + border: 0; + border-radius: 10px; + background: var(--sgdb-well); + box-shadow: var(--sgdb-well-inset); +} +.sgd-builder-stepper-field > div:first-child { display: grid; gap: 2px; } +.sgd-builder-stepper-field span { font-size: 12px; font-weight: 500; } +.sgd-builder-stepper-field small { color: var(--sgdb-muted); font-size: 10px; } +/* One integrated control: a single raised track with a hairline between the + two halves, rather than two separately outlined 34px blocks. */ +.sgd-builder-stepper { + display: grid; + grid-template-columns: 28px 40px 28px; + align-items: center; + border-radius: 8px; + background: var(--sgdb-thumb-bg); + box-shadow: var(--sgdb-thumb-shadow); +} +.sgd-builder-stepper button, .sgd-builder-request-stepper button { + min-width: 28px; + min-height: 30px; + padding: 0; + border: 0; + border-radius: 8px; + background: transparent; + color: var(--sgdb-ink); + cursor: pointer; + font-size: 14px; + transition: background-color var(--sgdb-ease-dur) var(--sgdb-ease); +} +.sgd-builder-stepper input { + width: 40px; + height: 30px; + padding: 0; + border-right: 0.5px solid color-mix(in oklch, var(--sgdb-ink) 10%, transparent); + border-left: 0.5px solid color-mix(in oklch, var(--sgdb-ink) 10%, transparent); + border-top: 0; + border-bottom: 0; + border-radius: 0; + appearance: textfield; + background: transparent; + color: var(--sgdb-ink); + font-size: 12px; + font-variant-numeric: tabular-nums; + font-weight: 590; + line-height: 30px; + text-align: center; +} +.sgd-builder-stepper input::-webkit-inner-spin-button, +.sgd-builder-stepper input::-webkit-outer-spin-button { margin: 0; appearance: none; } +.sgd-builder-stepper button:disabled, .sgd-builder-request-stepper button:disabled { opacity: 0.32; cursor: not-allowed; } +.sgd-builder-resource-summary { margin: 8px 0 0; color: var(--sgdb-muted); font-size: 11px; line-height: 1.4; } + +.sgd-builder-topology-toggle { margin-top: 10px; } +.sgd-builder-text-action { + justify-self: start; + min-height: 32px; + padding: 0; + border: 0; + background: transparent; + color: var(--sgdb-muted); + cursor: pointer; + font-size: 11px; + font-weight: 500; +} +.sgd-builder-text-action:disabled { cursor: not-allowed; opacity: 0.4; } +.sgd-builder-advanced { margin-top: 12px; padding-top: 12px; border-top: 1px solid var(--sgdb-line); } +.sgd-builder-advanced > p { margin: 0 0 12px; color: var(--sgdb-muted); font-size: 11px; line-height: 1.5; } +.sgd-builder-topology-inputs { display: grid; gap: 8px; } +.sgd-builder-topology-inputs label { display: grid; gap: 5px; color: var(--sgdb-muted); font-size: 10.5px; } +.sgd-builder-topology-inputs select { + width: 100%; + min-height: 38px; + padding: 0 8px; + border: 1px solid var(--sgdb-line); + border-radius: 6px; + background: var(--sgdb-raised); + color: var(--sgdb-ink); + font-size: 12px; +} +.sgd-builder-messages { margin-top: 12px; padding: 8px 10px; border-radius: 6px; background: color-mix(in oklch, var(--sgdb-warn) 9%, transparent); color: color-mix(in oklch, var(--sgdb-warn) 75%, var(--sgdb-ink)); } +.sgd-builder-messages[data-state="error"] { background: color-mix(in oklch, var(--sgdb-error) 9%, transparent); color: color-mix(in oklch, var(--sgdb-error) 78%, var(--sgdb-ink)); } +.sgd-builder-messages p { margin: 0; font-size: 11px; line-height: 1.45; } +.sgd-builder-messages p + p { margin-top: 5px; } + +.sgd-builder-setting-layout, .sgd-builder-setting-list, .sgd-builder-setting-item { min-width: 0; } +.sgd-builder-setting-item { border-bottom: 1px solid var(--sgdb-line); } +.sgd-builder-setting-item:last-child { border-bottom: 0; } +.sgd-builder-setting-row { + display: grid; + grid-template-columns: minmax(76px, auto) minmax(0, 1fr) auto auto; + width: 100%; + min-height: 44px; + padding: 8px; + gap: 8px; + align-items: center; + border: 0; + border-bottom: 0; + border-radius: 8px; + background: transparent; + transition: background-color var(--sgdb-ease-dur) var(--sgdb-ease), + color var(--sgdb-ease-dur) var(--sgdb-ease); + color: var(--sgdb-ink); + cursor: pointer; + text-align: left; +} +.sgd-builder-setting-row > span:first-child { color: var(--sgdb-muted); font-size: 11px; } +.sgd-builder-setting-row strong { overflow: hidden; font-size: 12px; font-weight: 500; text-overflow: ellipsis; white-space: nowrap; } +.sgd-builder-setting-row small { color: var(--sgdb-ok); font-size: 10px; font-weight: 500; } +.sgd-builder-setting-row > span:last-child { color: var(--sgdb-muted); font-size: 17px; } +.sgd-builder-setting-row[data-active="true"] { background: color-mix(in oklch, var(--sgdb-serve) 4%, transparent); color: var(--sgdb-serve); } + +.sgd-builder-context { padding: 16px 0 4px; } +.sgd-builder-context--inline { padding: 12px 4px 16px; border-top: 1px solid var(--sgdb-line); } +.sgd-builder-context--rail { display: none; } +.sgd-builder-context-heading { display: flex; align-items: start; justify-content: space-between; gap: 12px; } +.sgd-builder-context-heading > div > span { font-size: 13px; font-weight: 590; } +.sgd-builder-context-heading p { margin: 4px 0 0; color: var(--sgdb-muted); font-size: 11px; line-height: 1.45; } +.sgd-builder-context-heading > small { flex: none; color: var(--sgdb-muted); font-size: 10px; } +.sgd-builder-context-options { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 8px; margin-top: 12px; } +.sgd-builder-context-note { margin-top: 12px; padding-top: 10px; border-top: 1px solid var(--sgdb-line); } +.sgd-builder-context-note p { margin: 0; color: var(--sgdb-muted); font-size: 11px; line-height: 1.5; } +.sgd-builder-context-note a { + display: inline-flex; + margin-top: 8px; + min-height: 24px; + align-items: center; + gap: 5px; + color: var(--sgdb-serve); + font-size: 11px; + font-weight: 500; + text-decoration: none; + transition: color var(--sgdb-ease-dur) var(--sgdb-ease); +} +.sgd-builder-context-note a + a { margin-left: 16px; } +.sgd-builder-context-note a svg { flex: none; transition: transform var(--sgdb-ease-dur) var(--sgdb-ease); } +/* Hover answers: the label underlines and the glyph leans in. */ +.sgd-builder-context-note a:hover { + text-decoration: underline; + text-decoration-thickness: 1px; + text-underline-offset: 3px; +} +.sgd-builder-context-note a:hover svg { transform: translateX(1px) scale(1.08); } +.sgd-builder-scope-panel[data-scope="request"] .sgd-builder-context-note a { color: var(--sgdb-request); } +.sgd-builder-request-stepper { display: grid; grid-template-columns: 40px 52px 40px auto; gap: 6px; align-items: center; margin-top: 14px; } +.sgd-builder-request-stepper input { + width: 52px; + min-width: 0; + height: 34px; + padding: 0 4px; + border: 1px solid var(--sgdb-line); + border-radius: 6px; + appearance: textfield; + background: var(--sgdb-surface); + color: var(--sgdb-ink); + font-size: 16px; + font-variant-numeric: tabular-nums; + font-weight: 590; + text-align: center; +} +.sgd-builder-request-stepper input::-webkit-inner-spin-button, +.sgd-builder-request-stepper input::-webkit-outer-spin-button { margin: 0; appearance: none; } +.sgd-builder-request-stepper > span { margin-left: 4px; color: var(--sgdb-muted); font-size: 11px; } +.sgd-builder-request-setting { border-bottom: 1px solid var(--sgdb-line); } +.sgd-builder-request-setting:last-child { border-bottom: 0; } +.sgd-builder-request-setting .sgd-builder-context { padding: 16px 0; } + +/* stretch + the code area flexing below keep side-by-side cards the same + height, so their footers sit on one line instead of ragged bottoms. */ +.sgd-builder-output-rail { display: grid; gap: 12px; padding: 0 12px 16px; align-items: stretch; } +.sgd-builder-output { + --output-accent: var(--sgdb-serve); + display: flex; + min-width: 0; + flex-direction: column; + overflow: hidden; + border: 1px solid var(--sgdb-line); + border-radius: 12px; + background: var(--sgdb-raised); + box-shadow: 0 1px 2px oklch(30% 0.03 258 / 0.05); + transition: border-color 160ms ease-out, transform 160ms ease-out; +} +.sgd-builder-output[data-output="request"] { --output-accent: var(--sgdb-request); } +.sgd-builder-output[data-emphasized="true"] { border-color: color-mix(in oklch, var(--output-accent) 24%, var(--sgdb-line)); } +.sgd-builder-output > header { display: grid; grid-template-columns: 24px minmax(0, 1fr) auto; gap: 8px; align-items: center; min-height: 50px; padding: 8px 10px; border-bottom: 1px solid var(--sgdb-line); } +.sgd-builder-output[data-emphasized="true"] > header { + background: linear-gradient(180deg, + color-mix(in oklch, var(--output-accent) 7%, transparent), + color-mix(in oklch, var(--output-accent) 2%, transparent)); +} +/* Neutral step disc — the accents stay reserved for state (active tab, status), + so a static ordinal does not compete with them. */ +.sgd-builder-output-index { display: grid; width: 20px; height: 20px; place-items: center; border: 0; border-radius: 50%; background: var(--sgdb-well); color: var(--sgdb-muted); font-size: 10px; font-weight: 590; font-variant-numeric: tabular-nums; } +.sgd-builder-output-title { display: flex; min-width: 0; gap: 7px; align-items: baseline; } +.sgd-builder-output-title strong { font-size: 13px; font-weight: 590; } +.sgd-builder-output-title span { color: var(--sgdb-muted); font-size: 10.5px; } +.sgd-builder-status { display: inline-flex; gap: 5px; align-items: center; color: var(--sgdb-muted); font-size: 10px; white-space: nowrap; } +.sgd-builder-status > span { + width: 6px; + height: 6px; + border-radius: 50%; + background: var(--sgdb-warn); + box-shadow: 0 0 6px color-mix(in oklch, var(--sgdb-warn) 60%, transparent); +} +.sgd-builder-status[data-status="verified"] > span { background: var(--sgdb-ok); } +.sgd-builder-status[data-status="error"] > span { background: var(--sgdb-error); } +.sgd-builder-output-tabs { display: flex; min-height: 38px; padding: 5px 10px; gap: 14px; align-items: center; border-bottom: 1px solid var(--sgdb-line); } +.sgd-builder-output-tabs button { min-height: 28px; padding: 0; border: 0; border-bottom: 1px solid transparent; background: transparent; color: var(--sgdb-muted); cursor: pointer; font-size: 10.5px; font-weight: 500; } +.sgd-builder-output-tabs button[data-selected="true"] { border-bottom-color: var(--output-accent); color: var(--sgdb-ink); } +.sgd-builder-node-fields { display: grid; grid-template-columns: minmax(0, 1fr) 84px; gap: 8px; padding: 8px 10px; border-bottom: 1px solid var(--sgdb-line); } +.sgd-builder-node-fields label { display: grid; min-width: 0; gap: 4px; color: var(--sgdb-muted); font-size: 10px; } +.sgd-builder-node-fields input { width: 100%; min-width: 0; min-height: 34px; padding: 0 7px; border: 1px solid var(--sgdb-line); border-radius: 6px; background: var(--sgdb-surface); color: var(--sgdb-ink); font-size: 11px; } +/* Positioning context for the floating copy control; grid so the pre keeps + min-width:0 scrolling. flex:1 lets the code area absorb the height + difference between side-by-side cards. */ +.sgd-builder-code { position: relative; display: grid; min-width: 0; flex: 1 1 auto; } +.sgd-builder-code > pre { + height: 100%; + max-height: calc(13.95em + 24px); + min-width: 0; + margin: 0; + padding: 12px; + overflow: auto; + border: 0; + border-radius: 0; + background: var(--sgdb-code-bg); + box-shadow: inset 0 1px 3px var(--sgdb-code-inset); + color: var(--sgdb-ink); + font-size: clamp(11.5px, 1.2cqi, 12.5px); + line-height: 1.55; + white-space: pre; +} +.sgd-builder-code > pre::-webkit-scrollbar-corner { background: transparent; } +.sgd-builder-code > pre.is-expanded { height: auto; max-height: 70vh; } +.sgd-builder-output code { background: transparent; color: inherit; font: inherit; font-variant-ligatures: none; } +.sgd-builder-output-error { padding: 8px 10px; border-top: 1px solid color-mix(in oklch, var(--sgdb-error) 35%, var(--sgdb-line)); background: color-mix(in oklch, var(--sgdb-error) 8%, transparent); color: var(--sgdb-error); font-size: 10.5px; line-height: 1.4; } +.sgd-builder-output > footer { display: flex; min-height: 44px; padding: 5px 10px; align-items: center; justify-content: space-between; border-top: 1px solid var(--sgdb-line); } +.sgd-builder-output > footer > div { display: flex; margin-left: auto; gap: 12px; align-items: center; } +/* Copy floats inside the code area (top-right), revealed on hover — the + standard code-block pattern. A frosted chip keeps the glyph legible over + scrolled code without drawing a box when idle. */ +.sgd-builder-copy { + position: absolute; + top: 7px; + right: 7px; + display: grid; + width: 28px; + height: 28px; + padding: 0; + border: 0; + border-radius: 7px; + background: var(--sgdb-thumb-bg); + -webkit-backdrop-filter: blur(8px) saturate(1.4); + backdrop-filter: blur(8px) saturate(1.4); + box-shadow: var(--sgdb-thumb-shadow); + color: var(--sgdb-muted); + cursor: pointer; + opacity: 0; + place-items: center; + transform: translateY(2px) scale(0.9); + transition: opacity var(--sgdb-ease-dur) var(--sgdb-ease), + color var(--sgdb-ease-dur) var(--sgdb-ease), + transform var(--sgdb-spring-dur) var(--sgdb-spring); +} +.sgd-builder-code:hover > .sgd-builder-copy, +.sgd-builder-copy:focus-visible, +.sgd-builder-copy[data-copied="true"] { opacity: 1; transform: translateY(0) scale(1); } +/* No hover on touch — the control must stay visible. */ +@media (hover: none) { + .sgd-builder-copy { opacity: 1; transform: none; } +} +.sgd-builder-copy:disabled { cursor: not-allowed; color: color-mix(in oklch, var(--sgdb-muted) 45%, transparent); } +/* Both glyphs share the cell; the swap is a settle-in scale rather than a + hard display switch. */ +.sgd-builder-copy svg { + grid-area: 1 / 1; + pointer-events: none; + transition: opacity 140ms var(--sgdb-ease), transform 220ms var(--sgdb-ease); +} +.sgd-builder-copy .sgd-builder-copy-check { opacity: 0; transform: scale(0.5); } +.sgd-builder-copy[data-copied="true"] { color: var(--sgdb-ok); } +.sgd-builder-copy[data-copied="true"] .sgd-builder-copy-glyph { opacity: 0; transform: scale(0.5); } +.sgd-builder-copy[data-copied="true"] .sgd-builder-copy-check { opacity: 1; transform: scale(1); } + +/* Hover only changes tone. Lifting the card on hover (translateY) made the + whole grid twitch as the pointer crossed it. */ +@media (hover: hover) and (pointer: fine) { + .sgd-command-builder button:not(:disabled):not([data-blocked="true"]):hover { color: var(--sgdb-ink); } + .sgd-builder-scope-tabs button:not(:disabled):hover { color: var(--sgdb-ink); } + .sgd-builder-hardware:not(:disabled):not([data-selected="true"]):hover, + .sgd-builder-choice:not(:disabled):not([data-blocked="true"]):not([data-selected="true"]):hover { + background: var(--sgdb-pick-hover); + } + .sgd-builder-setting-row:not(:disabled):hover { background: color-mix(in oklch, var(--sgdb-serve) 5%, transparent); } + .sgd-builder-stepper button:not(:disabled):hover, + .sgd-builder-request-stepper button:not(:disabled):hover { background: var(--sgdb-pick-hover); } + .sgd-builder-copy:not(:disabled):hover { color: var(--sgdb-ink); } + .sgd-builder-copy[data-copied="true"]:hover { color: var(--sgdb-ok); } + .sgd-builder-text-action:not(:disabled):hover { color: var(--output-accent, var(--sgdb-brand)); } +} + +.sgd-command-builder :where(button, input, select, a):focus-visible { outline: 2px solid color-mix(in oklch, var(--sgdb-brand) 76%, white); outline-offset: 2px; } +/* Press reads as the control yielding under the finger, not jumping. */ +.sgd-command-builder button:not(:disabled):not([data-blocked="true"]):active { transform: scale(0.97); transition-duration: 60ms; } + +@container builder-controls (min-width: 560px) { + .sgd-builder-scope-panel[data-scope="base"] { grid-template-columns: repeat(2, minmax(0, 1fr)); column-gap: 28px; } + .sgd-builder-scope-panel[data-scope="base"] > .sgd-builder-recipe { grid-column: 1 / -1; } + /* Half a column cannot host two steppers side by side — stack them. */ + .sgd-builder-scope-panel[data-scope="base"] .sgd-builder-resource-grid { grid-template-columns: minmax(0, 1fr); } + .sgd-builder-setting-layout { display: grid; grid-template-columns: minmax(240px, 0.9fr) minmax(0, 1.1fr); } + .sgd-builder-setting-list { padding-right: 16px; border-right: 1px solid var(--sgdb-line); } + .sgd-builder-context--inline { display: none; } + .sgd-builder-context--rail { display: block; padding: 12px 0 4px 16px; } + .sgd-builder-request-direct { grid-template-columns: repeat(2, minmax(0, 1fr)); } + .sgd-builder-request-setting { padding-right: 16px; border-right: 1px solid var(--sgdb-line); border-bottom: 0; } + .sgd-builder-request-setting + .sgd-builder-request-setting { padding-right: 0; padding-left: 16px; border-right: 0; } +} + +/* Container queries measure the content box; subtract the two 1px borders so + the documented 420 / 720 / 960px outer widths switch at the intended point. */ +@container command-builder (max-width: 717px) { + .sgd-builder-output-rail { display: contents; } + .sgd-builder-controls { order: 0; } + .sgd-builder-output { width: calc(100% - 24px); margin: 12px 12px 16px; order: 1; } + .sgd-builder-main[data-scope="serve"] .sgd-builder-output[data-output="serve"], + .sgd-builder-main[data-scope="request"] .sgd-builder-output[data-output="request"] { order: -1; } +} + +@container command-builder (min-width: 718px) and (max-width: 957px) { + .sgd-builder-output-rail { order: -1; padding-top: 12px; } + .sgd-builder-main[data-scope="base"] .sgd-builder-output-rail { order: 1; padding-top: 0; } + .sgd-builder-output[data-emphasized="false"] { order: 1; } +} + +@container command-builder (min-width: 418px) { + .sgd-builder-scope-tabs { margin-inline: 16px; } + .sgd-builder-scope-tabs button { font-size: 13px; } + .sgd-builder-hardware-grid { grid-template-columns: repeat(3, minmax(0, 1fr)); } + .sgd-builder-resource-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); } + .sgd-builder-topology-inputs { grid-template-columns: repeat(3, minmax(0, 1fr)); } +} + +@container command-builder (min-width: 718px) { + .sgd-builder-scope-panel[data-scope="base"] { grid-template-columns: repeat(12, minmax(0, 1fr)); column-gap: 16px; } + .sgd-builder-scope-panel[data-scope="base"] > :where(.sgd-builder-recipe, .sgd-builder-section) { grid-column: 1 / -1; } + .sgd-builder-scope-panel[data-scope="base"] > .sgd-builder-section:nth-last-child(-n + 2) { grid-column: span 6; } + .sgd-builder-output-rail { grid-template-columns: minmax(0, 1fr); } + .sgd-builder-main[data-scope="base"] .sgd-builder-output-rail { grid-template-columns: repeat(2, minmax(0, 1fr)); } + .sgd-builder-choice-grid[data-density="compact"] { grid-template-columns: repeat(3, minmax(0, 1fr)); } +} + +@container command-builder (min-width: 958px) { + .sgd-builder-scope-tabs { margin-inline: 20px; } + .sgd-builder-scope-tabs button { font-size: 14px; } + .sgd-builder-main { display: grid; grid-template-columns: minmax(0, 38fr) minmax(0, 62fr); } + .sgd-builder-controls { padding: 8px 20px 24px; border-right: 1px solid var(--sgdb-line); } + .sgd-builder-output-rail { grid-template-columns: minmax(0, 1fr); padding: 16px; } + .sgd-builder-scope-panel[data-scope="base"] { display: grid; grid-template-columns: minmax(0, 1fr); } + .sgd-builder-scope-panel[data-scope="base"] > .sgd-builder-recipe, + .sgd-builder-scope-panel[data-scope="base"] > .sgd-builder-section, + .sgd-builder-scope-panel[data-scope="base"] > .sgd-builder-section:nth-last-child(-n + 2) { grid-column: 1; } + .sgd-builder-hardware-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); } + .sgd-builder-output { box-shadow: 0 1px 2px color-mix(in oklch, var(--sgdb-ink) 5%, transparent); } +} + +html.dark .sgd-command-builder, [data-theme="dark"] .sgd-command-builder { + --sgdb-surface: oklch(24% 0.015 258); + --sgdb-raised: oklch(27% 0.015 258); + --sgdb-ink: oklch(94% 0.006 80); + --sgdb-muted: oklch(71% 0.012 258); + --sgdb-line: rgba(255, 255, 255, 0.09); + --sgdb-brand: oklch(70% 0.14 40); + --sgdb-serve: oklch(74% 0.11 45); + --sgdb-request: oklch(77% 0.1 64); + --sgdb-ok: oklch(74% 0.11 153); + --sgdb-warn: oklch(76% 0.11 78); + --sgdb-error: oklch(73% 0.14 25); + /* Dark glass is built from white overlays: the thumb is a translucent + gradient with a specular top edge, wells recess into real black, and + the panel carries its own top highlight so it reads as a lit slab + rather than a flat grey rectangle on a near-identical page. */ + --sgdb-well: rgba(0, 0, 0, 0.32); + --sgdb-well-inset: inset 0 1px 2px rgba(0, 0, 0, 0.4); + --sgdb-thumb-bg: linear-gradient(180deg, rgba(255, 255, 255, 0.14), rgba(255, 255, 255, 0.07)); + --sgdb-thumb-shadow: 0 0 0 0.5px rgba(255, 255, 255, 0.16), + inset 0 1px 0 rgba(255, 255, 255, 0.13), + 0 2px 8px rgba(0, 0, 0, 0.35); + --sgdb-pick-hover: rgba(255, 255, 255, 0.06); + --sgdb-code-bg: oklch(18% 0.012 258); + --sgdb-code-inset: rgba(0, 0, 0, 0.5); + box-shadow: inset 0 1px 0 rgba(255, 255, 255, 0.06), + 0 16px 40px -16px rgba(0, 0, 0, 0.5); +} + +@media (prefers-reduced-motion: reduce) { + .sgd-command-builder *, .sgd-command-builder *::before, .sgd-command-builder *::after { scroll-behavior: auto !important; transition: none !important; animation: none !important; } +} + + .sgd-muted { color: rgb(107, 114, 128); font-size: 0.85rem; @@ -219,6 +876,13 @@ html.dark .sgd-chip, color: rgb(253, 186, 116); } +html.dark .sgd-model-tags .sgd-chip, +[data-theme="dark"] .sgd-model-tags .sgd-chip { + border: 0; + background: rgba(129, 140, 248, 0.18); + color: rgb(199, 210, 254); +} + html.dark .sgd-muted, [data-theme="dark"] .sgd-muted { color: rgb(156, 163, 175); diff --git a/docs/docs/sglang-diffusion/quantization.mdx b/docs/docs/sglang-diffusion/quantization.mdx index e037905c2..a3b25bf80 100644 --- a/docs/docs/sglang-diffusion/quantization.mdx +++ b/docs/docs/sglang-diffusion/quantization.mdx @@ -284,7 +284,7 @@ sglang generate \ MiniMax-H3 supports this path while preserving its required FP32 patch, timestep, and output projections. See the -[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-runtime-feature-recipes) +[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-feature-contracts-and-advanced-recipes) for its distributed serving recipe. ### MXFP4 Online Quantization @@ -334,7 +334,7 @@ projections take that path. `kitchen_int8` is approximate and is not a consistency ground-truth mode. The BF16 path is unchanged when `comfy-kitchen` is not installed. See the -[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-runtime-feature-recipes) +[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-feature-contracts-and-advanced-recipes) for the 24 GB offload recipe, including why `vae` must stay out of `--layerwise-offload-components`. diff --git a/docs/scripts/check_cookbook_configs.mjs b/docs/scripts/check_cookbook_configs.mjs index 35b610687..54e23eeb7 100755 --- a/docs/scripts/check_cookbook_configs.mjs +++ b/docs/scripts/check_cookbook_configs.mjs @@ -21,11 +21,12 @@ // role can remove that algorithm from the generated command. import { readFileSync, readdirSync } from "node:fs"; -import { dirname, join, relative } from "node:path"; +import { basename, dirname, join, relative } from "node:path"; import { fileURLToPath } from "node:url"; const SNIPPETS = join(dirname(fileURLToPath(import.meta.url)), "..", "src", "snippets"); const CONFIGS = join(SNIPPETS, "configs"); +const DIFFUSION_COOKBOOK = join(SNIPPETS, "..", "..", "cookbook", "diffusion"); const LEGACY_DIMS = ["variants", "quantizations", "strategies", "nodesOptions"]; const failures = []; @@ -88,11 +89,14 @@ const selectionSpace = (config) => { ...(config.matchDims || []), ...(config.overlayDims || []), ]; - let space = [{}]; + let space = [{ ...(config.commandBuilder?.defaultSelection || {}) }]; for (const d of dims) { const next = []; + const options = d.kind === "number" + ? [...new Set([d.default, d.min, d.max])].map((id) => ({ id })) + : (d.options || []); for (const partial of space) { - for (const opt of (d.options || [])) next.push({ ...partial, [d.id]: opt.id }); + for (const opt of options) next.push({ ...partial, [d.id]: opt.id }); } space = next.length ? next : space; if (space.length > 20000) return space.slice(0, 20000); // cheap blow-up guard @@ -154,11 +158,144 @@ for (const path of walk(CONFIGS)) { for (const dim of (config.overlayDims || [])) { const ids = (dim.options || []).map((o) => o.id); - if (dim.default !== undefined && !ids.includes(dim.default)) { + if (dim.kind === "number") { + if (!Number.isInteger(dim.min) || !Number.isInteger(dim.max) || dim.min > dim.max) { + fail(where, `overlayDims.${dim.id} has invalid numeric bounds`); + } + if (!Number.isInteger(dim.default) || dim.default < dim.min || dim.default > dim.max) { + fail(where, `overlayDims.${dim.id}.default is outside [${dim.min}, ${dim.max}]`); + } + } else if (dim.default !== undefined && !ids.includes(dim.default)) { fail(where, `overlayDims.${dim.id}.default="${dim.default}" is not one of [${ids}]`); } } + const builder = config.commandBuilder; + if (builder) { + const scopes = new Set(["base", "serve", "request"]); + for (const dim of (config.overlayDims || [])) { + if (!scopes.has(dim.scope)) { + fail(where, `builder dimension ${dim.id} has invalid scope "${dim.scope}"`); + } + } + if (!builder.defaultSelection || typeof builder.defaultSelection !== "object") { + fail(where, "commandBuilder.defaultSelection is required"); + } + if (typeof builder.resource?.autoTopology !== "function") { + fail(where, "commandBuilder.resource.autoTopology must be a function"); + } + if (typeof builder.resource?.validateTopology !== "function") { + fail(where, "commandBuilder.resource.validateTopology must be a function"); + } + if (typeof builder.resolveDeployment !== "function") { + fail(where, "commandBuilder.resolveDeployment must be a function"); + } + for (const [key, bounds] of Object.entries(builder.resource?.limits || {})) { + if (!Number.isInteger(bounds.min) || !Number.isInteger(bounds.max) || bounds.min > bounds.max) { + fail(where, `commandBuilder.resource.limits.${key} is invalid`); + } + const value = builder.defaultSelection?.[key]; + if (!Number.isInteger(value) || value < bounds.min || value > bounds.max) { + fail(where, `commandBuilder.defaultSelection.${key} is outside its bounds`); + } + } + + const selectionOf = (extra = {}) => { + const defaults = { ...(builder.defaultSelection || {}) }; + for (const dim of (config.overlayDims || [])) defaults[dim.id] = dim.default; + return { ...defaults, ...extra }; + }; + const validateResolved = (selection, label, expectVerified = false) => { + let resolved; + try { + resolved = builder.resolveDeployment(selection); + } catch (e) { + fail(where, `${label} resolver throws: ${e.message}`); + return null; + } + if (!resolved || !Array.isArray(resolved.flags) || !resolved.builder) { + fail(where, `${label} resolver must return a cell with flags and builder metadata`); + return resolved; + } + if (!Array.isArray(resolved.builder.errors) || !Array.isArray(resolved.builder.warnings)) { + fail(where, `${label} resolver errors/warnings must be arrays`); + } + if (expectVerified && (resolved.builder.errors?.length || resolved.builder.verification?.serve !== "verified")) { + fail(where, `${label} is declared verified but resolved as ${resolved.builder.verification?.serve || "invalid"}`); + } + + let flags = [...resolved.flags]; + for (const dim of (config.overlayDims || [])) { + const option = (dim.options || []).find((entry) => entry.id === selection[dim.id]); + if (!option) continue; + const strip = typeof option.stripPrefixes === "function" + ? option.stripPrefixes(selection) : (option.stripPrefixes || []); + if (strip.length) flags = flags.filter((flag) => !strip.includes(flag.split(/[\s=]/)[0])); + const extra = typeof option.flags === "function" ? option.flags(selection) : option.flags; + flags.push(...(extra || [])); + } + const families = flags.map((flag) => flag.split(/[\s=]/)[0]); + const duplicate = families.find((family, index) => families.indexOf(family) !== index); + if (duplicate) fail(where, `${label} emits duplicate flag family ${duplicate}`); + return resolved; + }; + + validateResolved(selectionOf(), "commandBuilder default"); + const recipeIds = new Set(); + const recipeSignatures = new Set(); + for (const [index, recipe] of (builder.resource?.verifiedRecipes || []).entries()) { + if (!recipe.id || recipeIds.has(recipe.id)) fail(where, `verifiedRecipes[${index}] has a duplicate/missing id`); + recipeIds.add(recipe.id); + const signature = [recipe.hw, recipe.nodes, recipe.gpus_per_node, recipe.placement, + recipe.tp_size, recipe.ulysses_degree, recipe.ring_degree].join("|"); + if (recipeSignatures.has(signature)) fail(where, `verifiedRecipes[${index}] duplicates ${signature}`); + recipeSignatures.add(signature); + const selection = selectionOf({ ...recipe, topology_mode: "auto" }); + const topology = builder.resource.autoTopology(selection); + const errors = builder.resource.validateTopology(selection, topology); + if (!Array.isArray(errors) || errors.length) { + fail(where, `verifiedRecipes[${index}] fails topology validation: ${(errors || []).join("; ")}`); + } + validateResolved(selection, `verifiedRecipes[${index}]`, true); + } + + // H3's architectural contract is important enough to pin directly: exact + // platform recipes, legal custom admission, and each invalidity family. + if (config.modelName === "MiniMax-H3") { + const checkH3 = (label, extra, expected, verified = true) => { + const selection = selectionOf(extra); + const resolved = validateResolved(selection, `H3 ${label}`, verified); + if (!resolved) return; + for (const [key, value] of Object.entries(expected)) { + if (resolved.builder.topology?.[key] !== value) { + fail(where, `H3 ${label} topology.${key}=${resolved.builder.topology?.[key]}, expected ${value}`); + } + } + }; + checkH3("B200 1x8", { hw: "b200", nodes: 1, gpus_per_node: 8, placement: "resident" }, { tp_size: 1, ulysses_degree: 8, ring_degree: 1 }); + checkH3("H100 1x4", { hw: "h100", nodes: 1, gpus_per_node: 4, placement: "resident" }, { tp_size: 2, ulysses_degree: 2, ring_degree: 1 }); + checkH3("H200 2x8", { hw: "h200", nodes: 2, gpus_per_node: 8, placement: "resident" }, { tp_size: 1, ulysses_degree: 8, ring_degree: 2 }); + for (const hw of ["mi300x", "mi355x"]) { + for (const count of [1, 2, 4, 8]) { + checkH3(`${hw} 1x${count}`, { hw, nodes: 1, gpus_per_node: count, placement: "resident" }, { tp_size: 1, ulysses_degree: count, ring_degree: 1 }); + } + } + const custom = validateResolved(selectionOf({ hw: "b200", nodes: 1, gpus_per_node: 2, placement: "resident" }), "H3 legal custom"); + if (custom?.builder.errors?.length || custom?.builder.verification?.serve !== "unverified") { + fail(where, "H3 legal custom topology must be copyable and Unverified"); + } + for (const [label, extra] of [ + ["3 GPU", { hw: "h200", nodes: 1, gpus_per_node: 3, placement: "resident" }], + ["TP3", { hw: "h100", nodes: 1, gpus_per_node: 4, placement: "resident", topology_mode: "manual", tp_size: 3, ulysses_degree: 1, ring_degree: 1 }], + ["head divisibility", { hw: "h200", nodes: 2, gpus_per_node: 8, placement: "resident", topology_mode: "manual", tp_size: 2, ulysses_degree: 8, ring_degree: 1 }], + ["sequence alignment", { hw: "mi300x", nodes: 2, gpus_per_node: 3, placement: "resident", topology_mode: "manual", tp_size: 1, ulysses_degree: 2, ring_degree: 3 }], + ]) { + const resolved = validateResolved(selectionOf(extra), `H3 invalid ${label}`); + if (!resolved?.builder.errors?.length) fail(where, `H3 invalid ${label} was not rejected`); + } + } + } + // Predicates and flag builders must survive every reachable selection. const space = selectionSpace(config); const probe = (fn, label) => { @@ -185,10 +322,13 @@ for (const path of walk(CONFIGS)) { } for (const dim of [...(config.matchDims || []), ...(config.overlayDims || [])]) { if (typeof dim.showWhen === "function") probe(dim.showWhen, `${dim.id}.showWhen`); + if (typeof dim.verifiedWhen === "function") probe(dim.verifiedWhen, `${dim.id}.verifiedWhen`); for (const opt of (dim.options || [])) { const tag = `${dim.id}.${opt.id}`; if (typeof opt.showWhen === "function") probe(opt.showWhen, `${tag}.showWhen`); if (typeof opt.disabled === "function") probe(opt.disabled, `${tag}.disabled`); + if (typeof opt.soft === "function") probe(opt.soft, `${tag}.soft`); + if (typeof opt.verifiedWhen === "function") probe(opt.verifiedWhen, `${tag}.verifiedWhen`); for (const key of ["flags", "env", "hints"]) { if (typeof opt[key] !== "function") continue; probe((sel) => { @@ -231,6 +371,81 @@ for (const path of walk(CONFIGS)) { } } +// ----------------------------------------------------- Diffusion page opening +// Keep the first screen consistent across model pages. This check intentionally +// guards structure, not editorial judgment; the authoring skill carries the +// capability/strength/boundary rubric that cannot be reduced to keywords. +const walkMdx = (dir) => readdirSync(dir, { withFileTypes: true }).flatMap((e) => + e.isDirectory() ? walkMdx(join(dir, e.name)) + : (e.name.endsWith(".mdx") ? [join(dir, e.name)] : [])); + +for (const path of walkMdx(DIFFUSION_COOKBOOK)) { + if (["README.mdx", "intro.mdx"].includes(basename(path))) continue; + + const where = relative(join(SNIPPETS, "..", ".."), path); + const src = readFileSync(path, "utf8"); + const quickStart = /^## 1\. Quick start\s*$/m.exec(src); + const legacyIntroduction = /^## 1\. Model Introduction\s*$/m.exec(src); + const sectionOne = quickStart || legacyIntroduction; + if (!sectionOne) { + fail(where, "missing `## 1. Quick start` or legacy `## 1. Model Introduction` heading"); + continue; + } + + const capabilityHeading = quickStart + ? /^## 2\. Model capabilities\s*$/m.exec(src) + : legacyIntroduction; + if (!capabilityHeading) { + fail(where, "Quick start pages need an exact `## 2. Model capabilities` heading"); + continue; + } + + const importPattern = /import\s+\{\s*DiffusionModelTags\s*\}\s+from\s+['"]\/src\/snippets\/diffusion\/model-tags\.jsx['"]/; + if (!importPattern.test(src.slice(0, sectionOne.index))) { + fail(where, "missing the shared DiffusionModelTags import before section 1"); + } + + const tagMatch = //.exec(src.slice(0, sectionOne.index)); + if (!tagMatch) { + fail(where, "missing `` before section 1"); + } else { + const tags = [...tagMatch[1].matchAll(/["']([^"']+)["']/g)].map((m) => m[1].trim()); + if (tags.length < 4 || tags.length > 6) { + fail(where, `tag widget has ${tags.length} tags; expected 4–6`); + } + if (tags.some((tag) => !tag)) fail(where, "tag widget contains an empty tag"); + } + + if (quickStart) { + const quickStartBody = src.slice(quickStart.index + quickStart[0].length, capabilityHeading.index); + if (!quickStartBody.includes("")) { + fail(where, "Quick start must render the command builder before model capabilities"); + } + if (!quickStartBody.includes('uv pip install "sglang[diffusion]"')) { + fail(where, "Quick start must include the diffusion installation command"); + } + } + + const introStart = capabilityHeading.index + capabilityHeading[0].length; + const rest = src.slice(introStart); + const boundaries = ["\n|", "\n rest.indexOf(marker)) + .filter((i) => i >= 0); + const lead = rest.slice(0, boundaries.length ? Math.min(...boundaries) : rest.length).trim(); + const paragraphs = lead.split(/\n\s*\n/).map((p) => p.trim()).filter(Boolean); + if (paragraphs.length < 2) { + fail(where, "model capability introduction needs at least two lead paragraphs"); + } + const prose = lead + .replace(/\[([^\]]+)]\([^)]+\)/g, "$1") + .replace(/`[^`]+`/g, "value") + .replace(/[*_#]/g, " "); + const words = prose.match(/[A-Za-z0-9][A-Za-z0-9+./@–—-]*/g) || []; + if (words.length < 45 || words.length > 180) { + fail(where, `lead introduction has ${words.length} words; expected 45–180`); + } +} + if (failures.length) { console.error(`FAIL (${failures.length})`); for (const f of failures) console.error(" - " + f); diff --git a/docs/src/snippets/_deployment.jsx b/docs/src/snippets/_deployment.jsx index d36a7c6ba..262266776 100644 --- a/docs/src/snippets/_deployment.jsx +++ b/docs/src/snippets/_deployment.jsx @@ -18,14 +18,33 @@ // `single` or `multi-N` → --nnodes N) // matchDims optional — replaces the legacy four. {id, title, options}[] // where each option is {id, label, showWhen?(sel), disabled?, -// disableReason?}. `hw` is always the implicit first dim. -// Cells are then keyed on (hw × ). +// disableReason?, soft?, softReason?}. `hw` is always the +// implicit first dim. Cells are then keyed on (hw × ). `disabled` is for combinations that cannot work; +// an option that runs but sits outside the verified matrix +// should declare `soft` instead — it stays selectable and +// announces itself as unverified (tooltip + in-cell note). +// Blocked options flash their disableReason under the row +// when tapped, so the reason also reaches touch readers. // overlayDims optional — rows that do NOT participate in cell lookup; the // picked option layers onto the matched cell, so an orthogonal // knob does not multiply the cell count. Same option shape plus // `flags` / `env` / `hints` (each a literal array or a function // of the whole selection), and a row-level `default` / `showWhen`. // `hints` render as `# ...` lines above the command. +// Builder-aware dimensions may additionally declare +// `scope: "base" | "serve" | "request"`, `description`, +// `quality`, `verifiedWhen`, and `learnMore`. Legacy configs +// omit these fields and keep the original renderer. +// commandBuilder optional — opts this config into the responsive diffusion +// builder while reusing this engine's overlay composition and +// command rendering. Shape: +// {defaultSelection, resource: {limits, verifiedRecipes, +// autoTopology(sel), validateTopology(sel)}, +// resolveDeployment(sel)}. The resolver returns a cell plus +// `builder` metadata (topologySummary, errors, warnings, +// verification, resolvedSettings). UI-only scope/expand and +// local head-address/rank state never enter the URL hash. // cells {match, verified?, verificationStatus?, nnodes?, warn?, redirect?, // env, flags}[] — one per // (hw × match dims); env/flags are flat literals, only @@ -474,6 +493,7 @@ export const Deployment = ({ config, benchmarks }) => { options: d.options || config[d.optionsKey] || [], })); const overlayDimSpecs = config.overlayDims || []; + const commandBuilder = config.commandBuilder || null; // DIMENSIONS is ordered by priority — higher-index dims adapt to lower-index // picks, never the reverse. Drives the grey-out/snap logic below. const DIMENSIONS = ["hw", ...matchDimSpecs.map((d) => d.id)]; @@ -545,6 +565,11 @@ export const Deployment = ({ config, benchmarks }) => { return out; }; // ==== end MIRROR ==== + // `soft` marks an option that is plausible but outside the verified matrix: + // it stays selectable (the status badge reports verification separately), + // where `disabled` is reserved for combinations that cannot work at all. + const optionSoft = (opt, sel) => + typeof opt.soft === "function" ? opt.soft(sel) : !!opt.soft; const findCell = (cells, sel) => cells.find((c) => DIMENSIONS.every((d) => c.match[d] === sel[d])); @@ -680,6 +705,8 @@ export const Deployment = ({ config, benchmarks }) => { // dim it is a property of the cell itself (`nnodes`), since the deployment // shape is then fixed by the hardware rather than picked by the reader. const parseNnodes = (id) => { + if (Number.isInteger(id)) return id; + if (/^\d+$/.test(id || "")) return parseInt(id, 10); if (id === "single") return 1; const m = /^multi-(\d+)$/.exec(id || ""); return m ? parseInt(m[1], 10) : 1; @@ -709,12 +736,14 @@ export const Deployment = ({ config, benchmarks }) => { if (multinode) { // Insert the multi-node trio after the last parallelism flag, // falling back to right after --model-path. - const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp-size", "--tp"]; - let i = -1; - for (const anchor of PARALLELISM_ANCHORS) { - i = flags.findIndex((f) => f.split(/[\s=]/)[0] === anchor); - if (i !== -1) break; - } + const PARALLELISM_ANCHORS = new Set([ + "--enable-dp-attention", "--dp", "--tp-size", "--tp", + "--sp-degree", "--ulysses-degree", "--ring-degree", + ]); + let i = flags.reduce( + (last, flag, index) => PARALLELISM_ANCHORS.has(flag.split(/[\s=]/)[0]) ? index : last, + -1, + ); if (i === -1) i = flags.findIndex((f) => f.startsWith("--model-path")); flags.splice(i + 1, 0, `--nnodes ${nnodes}`, @@ -1096,7 +1125,7 @@ export const Deployment = ({ config, benchmarks }) => { // verified cell first). Overlay dims seed from their own `default`, or the // first option, since no cell carries them. const initialSelectionFromCells = () => { - const first = config.cells[0]; + const first = (config.cells || [])[0]; const sel = Object.fromEntries( DIMENSIONS.map((d) => [d, first ? first.match[d] : ""]), ); @@ -1104,7 +1133,48 @@ export const Deployment = ({ config, benchmarks }) => { const opts = spec.options || []; sel[spec.id] = spec.default ?? (opts[0] && opts[0].id) ?? ""; } - return sel; + if (!commandBuilder) return sel; + return { + ...sel, + hw: commandBuilder.defaultSelection?.hw || config.supportedHardware?.[0] || "", + ...(commandBuilder.defaultSelection || {}), + }; + }; + + // Builder hashes are semantic rather than cell ids. Keep known dimension + // values, clamp bounded resources, and discard stale topology overrides. + // UI state (active scope, expanded cards, head IP, node rank) is deliberately + // absent from this object, so shared links stay portable and credential-free. + const normalizeBuilderSelection = (parsed) => { + const out = { ...initialSelectionFromCells(), ...parsed }; + if (!(config.supportedHardware || []).includes(out.hw)) { + out.hw = commandBuilder.defaultSelection?.hw || config.supportedHardware?.[0] || ""; + } + for (const spec of overlayDimSpecs) { + if (spec.kind === "number") { + const value = Number.parseInt(out[spec.id], 10); + out[spec.id] = Math.min( + spec.max, + Math.max(spec.min, Number.isFinite(value) ? value : Number(spec.default ?? spec.min)), + ); + continue; + } + const options = spec.options || []; + if (!options.some((option) => option.id === out[spec.id])) { + out[spec.id] = spec.default ?? options[0]?.id ?? ""; + } + } + for (const [key, bounds] of Object.entries(commandBuilder.resource?.limits || {})) { + const fallback = Number(commandBuilder.defaultSelection?.[key] ?? bounds.min ?? 1); + const value = Number.parseInt(out[key], 10); + out[key] = Math.min(bounds.max, Math.max(bounds.min, Number.isFinite(value) ? value : fallback)); + } + for (const key of ["tp_size", "ulysses_degree", "ring_degree"]) { + const value = Number.parseInt(out[key], 10); + out[key] = Number.isFinite(value) && value > 0 ? value : 1; + } + out.topology_mode = out.topology_mode === "manual" ? "manual" : "auto"; + return out; }; const placeholderDefaults = (schema) => { @@ -1164,8 +1234,13 @@ export const Deployment = ({ config, benchmarks }) => { if (key in parsed) { parsed[key] = value; touched = true; } }); if (!touched) return; - // Snap to a real cell if the hash named an impossible combo (stale link). - setSel(validateSelection(config.cells, parsed)); + // Cell configs snap to a real recipe; builders normalize their semantic + // resource state without forcing a custom-but-valid topology to a preset. + setSel( + commandBuilder + ? normalizeBuilderSelection(parsed) + : validateSelection(config.cells, parsed), + ); const historyState = window.history.state; const isInternalHash = historyState && @@ -1225,6 +1300,25 @@ export const Deployment = ({ config, benchmarks }) => { : config.runModes; const runModes = configuredRunModes || ["python", "docker"]; const [runMode, setRunMode] = useState(runModes[0]); // "python" | "docker" + const [builderScope, setBuilderScope] = useState("base"); + const [builderServerSetting, setBuilderServerSetting] = useState(null); + const [builderAdvanced, setBuilderAdvanced] = useState(false); + const [serveExpanded, setServeExpanded] = useState(false); + const [requestExpanded, setRequestExpanded] = useState(false); + const [builderHeadAddress, setBuilderHeadAddress] = useState(""); + const [builderNodeRank, setBuilderNodeRank] = useState(0); + // Tapping a disabled option surfaces its reason under the row — hover-only + // tooltips never reach touch readers. {dim, reason}; each note clears only + // itself, so a newer note is never cut short by an older timer. + const [blockedNote, setBlockedNote] = useState(null); + const flashBlockedNote = (dim, reason) => { + const note = { dim, reason }; + setBlockedNote(note); + setTimeout(() => setBlockedNote((cur) => (cur === note ? null : cur)), 4000); + }; + useEffect(() => { + if (builderNodeRank >= Number(sel.nodes || 1)) setBuilderNodeRank(0); + }, [sel.nodes, builderNodeRank]); const hasRunMode = runModes.includes(runMode); const fallbackRunMode = runModes[0]; const activeRunMode = hasRunMode ? runMode : fallbackRunMode; @@ -1248,7 +1342,10 @@ export const Deployment = ({ config, benchmarks }) => { // ==== 5. Derived values ==== const s = makeStyles(isDark); - const cell = findCell(config.cells, sel); + const cell = commandBuilder + ? commandBuilder.resolveDeployment(sel) + : findCell(config.cells, sel); + const builderMeta = (cell && cell.builder) || {}; const verifyStatus = cellVerifyStatus(cell, sel); // Pin the calculator-computed ratio into the rendered command (before the // host/port tail); cells themselves stay ratio-free. @@ -1262,7 +1359,14 @@ export const Deployment = ({ config, benchmarks }) => { else flags.push(line); return { ...cell, flags }; })(); - const command = renderCommand(cellWithRatio, sel, env, activeRunMode); + const commandEnv = commandBuilder + ? { + ...env, + NODE_RANK: String(builderNodeRank), + NODE0_IP: builderHeadAddress || "", + } + : env; + const command = renderCommand(cellWithRatio, sel, commandEnv, activeRunMode); // Speculative-decoding hint on the EFFECTIVE flags — speculation can arrive via // the Spec Decode overlay as well as the cell. SGLang resets // --max-running-requests to 48 when spec is on and it's unset; verified for both @@ -1347,7 +1451,8 @@ export const Deployment = ({ config, benchmarks }) => { const isEnabled = (dim, value) => { const opt = findOption(dim, value); if (opt && optionDisabled(opt, sel)) return false; - return isOverlayDim(dim) || isOptionAvailable(config.cells, sel, dim, value); + if (commandBuilder && dim === "hw") return true; + return isOverlayDim(dim) || isOptionAvailable(config.cells || [], sel, dim, value); }; // Switching a match dim can hide the option a dependent row currently holds @@ -1365,7 +1470,46 @@ export const Deployment = ({ config, benchmarks }) => { return out; }; + const recommendedBuilderRecipe = (hw) => { + const recipes = commandBuilder.resource?.verifiedRecipes || []; + return recipes.find((entry) => entry.hw === hw && entry.default) + || recipes.find((entry) => entry.hw === hw); + }; + const handleSelect = (dim, value) => { + if (commandBuilder) { + setSel((prev) => { + let next = { ...prev, [dim]: value }; + if (dim === "hw") { + const currentRecipe = recommendedBuilderRecipe(prev.hw); + const nextRecipe = recommendedBuilderRecipe(value); + const resourcesFollowPlatformDefault = !!currentRecipe + && Number(prev.nodes) === Number(currentRecipe.nodes) + && Number(prev.gpus_per_node) === Number(currentRecipe.gpus_per_node); + next = { + ...next, + nodes: resourcesFollowPlatformDefault + ? (nextRecipe?.nodes ?? next.nodes) + : next.nodes, + gpus_per_node: resourcesFollowPlatformDefault + ? (nextRecipe?.gpus_per_node ?? next.gpus_per_node) + : next.gpus_per_node, + topology_mode: "auto", + tp_size: resourcesFollowPlatformDefault + ? (nextRecipe?.tp_size ?? 1) + : next.tp_size, + ulysses_degree: resourcesFollowPlatformDefault + ? (nextRecipe?.ulysses_degree ?? 1) + : next.ulysses_degree, + ring_degree: resourcesFollowPlatformDefault + ? (nextRecipe?.ring_degree ?? 1) + : next.ring_degree, + }; + } + return reseatHiddenPicks(normalizeBuilderSelection(next)); + }); + return; + } setSel((prev) => reseatHiddenPicks( isOverlayDim(dim) @@ -1375,6 +1519,70 @@ export const Deployment = ({ config, benchmarks }) => { ); }; + const commitBuilderNumber = (event, currentValue, bounds, commit) => { + const parsed = Number(event.currentTarget.value); + if (!Number.isInteger(parsed)) { + event.currentTarget.value = String(currentValue); + return; + } + const value = Math.min(bounds.max, Math.max(bounds.min, parsed)); + event.currentTarget.value = String(value); + commit(value); + }; + + const renderBuilderNumberInput = ({ identity, value, min, max, label, onCommit }) => ( + event.currentTarget.select()} + onBlur={(event) => commitBuilderNumber( + event, + value, + { min, max }, + onCommit, + )} + onKeyDown={(event) => { + if (event.key === "Enter") event.currentTarget.blur(); + }} + /> + ); + + const updateBuilderResource = (key, delta) => { + if (!commandBuilder) return; + const bounds = commandBuilder.resource?.limits?.[key] || { min: 1, max: 8 }; + setSel((prev) => { + const value = Math.min(bounds.max, Math.max(bounds.min, Number(prev[key]) + delta)); + return normalizeBuilderSelection({ ...prev, [key]: value, topology_mode: "auto" }); + }); + }; + + const setBuilderResource = (key, rawValue) => { + if (!commandBuilder) return; + const value = Number.parseInt(rawValue, 10); + if (!Number.isFinite(value)) return; + const bounds = commandBuilder.resource?.limits?.[key] || { min: 1, max: 8 }; + setSel((prev) => normalizeBuilderSelection({ + ...prev, + [key]: Math.min(bounds.max, Math.max(bounds.min, value)), + topology_mode: "auto", + })); + }; + + const editBuilderTopology = (key, value) => { + if (!commandBuilder) return; + setSel((prev) => normalizeBuilderSelection({ + ...prev, + topology_mode: "manual", + [key]: Number.parseInt(value, 10) || 1, + })); + }; + const handleCopy = () => { navigator.clipboard.writeText(command); setCopied(true); @@ -1408,6 +1616,11 @@ export const Deployment = ({ config, benchmarks }) => { return (