[diffusion] feat: support LoRA for LTX2.3 (#23649)

2026-04-25 01:52:41 +08:00
parent baa0aa670f
commit 05696527ea
15 changed files with 810 additions and 209 deletions
@@ -0,0 +1,206 @@
+---
+title: LTX
+description: Run LTX-2 and LTX-2.3 video generation pipelines with SGLang Diffusion.
+metatags:
+    description: "Deploy and use LTX-2 and LTX-2.3 video generation models with SGLang Diffusion, including one-stage, two-stage, HQ, TI2V, and LoRA examples."
+---
+
+import { LTXDeployment } from '/src/snippets/diffusion/ltx-deployment.jsx';
+
+## 1. Model Introduction
+
+[LTX-2](https://huggingface.co/Lightricks/LTX-2) and [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) are video generation models from Lightricks. SGLang Diffusion supports the LTX series through native one-stage and two-stage pipelines for text-to-video and image-conditioned video generation.
+
+Use `Lightricks/LTX-2` or `Lightricks/LTX-2.3` as `--model-path`. For two-stage generation, SGLang uses the spatial upsampler and distilled LoRA components from the model snapshot by default. LTX-2.3 also supports the HQ two-stage variant.
+
+<Warning>
+**License notice:** LTX-2 and LTX-2.3 are released under the LTX-2 Community License Agreement, not Apache 2.0. The license includes commercial-use restrictions for some entities. Review the [official Lightricks license](https://huggingface.co/Lightricks/LTX-2.3/blob/main/LICENSE) before production or commercial use; SGLang support does not grant additional model usage rights.
+</Warning>
+
+## 2. SGLang-diffusion Installation
+
+Install SGLang with diffusion dependencies:
+
+```bash Command
+uv pip install "sglang[diffusion]" --prerelease=allow
+```
+
+For platform-specific setup, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation).
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different LTX pipelines and hardware targets.
+
+### 3.1 Basic Configuration
+
+The LTX series supports one-stage and two-stage pipelines. LTX-2.3 also supports the HQ two-stage pipeline. The recommended launch configuration depends on whether the target GPU can keep both two-stage DiTs resident.
+
+**Interactive Command Generator**: Use the configuration selector below to generate a deployment command. The default selection targets a single NVIDIA H200 with `resident` two-stage mode, which is the fastest startup path for the specified high-memory environment.
+
+<LTXDeployment />
+
+### 3.2 Configuration Tips
+
+Choose the pipeline class based on the quality and latency target:
+
+| Use case | Pipeline class | Notes |
+| --- | --- | --- |
+| One-stage generation | `LTX2Pipeline` | Fastest LTX native path. Supports T2V and TI2V. |
+| Two-stage generation | `LTX2TwoStagePipeline` | Uses a base stage and a refinement stage. Supported by LTX-2 and LTX-2.3. |
+| Two-stage High Quality (HQ) generation | `LTX2TwoStageHQPipeline` | LTX-2.3 HQ path; defaults to 1920x1088 unless you override `--width` and `--height`. |
+
+Feature compatibility:
+
+| Pipeline class | T2V | TI2V (`--image-path`) | LoRA (`--lora-path`) | Notes |
+| --- | --- | --- | --- | --- |
+| `LTX2Pipeline` | Yes | Yes | Yes | One-stage path. Cannot be combined with HQ because HQ is a separate two-stage pipeline class. |
+| `LTX2TwoStagePipeline` | Yes | Yes | Yes | Standard two-stage path for LTX-2 and LTX-2.3. |
+| `LTX2TwoStageHQPipeline` | Yes | Yes | Yes | High Quality two-stage path for LTX-2.3. Use this instead of `LTX2Pipeline`; it is not a one-stage mode flag. |
+
+For two-stage pipelines, `--ltx2-two-stage-device-mode` controls transformer residency:
+
+| Mode | When to use it |
+| --- | --- |
+| `snapshot` | Recommended default. Balances latency and VRAM. |
+| `resident` | Best latency on high-VRAM GPUs because both DiTs can stay resident. |
+| `original` | Closest to the original two-stage switching semantics. |
+
+Other deployment flags:
+
+- `--lora-path`: Preload a community LoRA adapter.
+- `--lora-weight-name`: Select the exact safetensors file when the LoRA repository contains multiple weight files.
+
+<Note>
+For native LTX-2.3 two-stage serving without a user LoRA, `resident` is the fastest high-VRAM path. When you pass `--lora-path`, SGLang still applies the user LoRA during the two-stage switch, so use `resident` on H200-class GPUs for enough VRAM, but do not expect the same premerged-stage2 benefit as the no-user-LoRA path.
+</Note>
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+The examples below spell out the current SGLang sampling defaults for reproducibility:
+
+| Model path | Default output | Default frames | Default steps |
+| --- | --- | --- | --- |
+| `Lightricks/LTX-2` | 768x512 | 121 | 40 |
+| `Lightricks/LTX-2.3` | 768x512 | 121 | 30 |
+| `Lightricks/LTX-2.3` with `LTX2TwoStageHQPipeline` | 1920x1088 | 121 | 15 |
+
+#### 4.1.1 LTX-2 one-stage text-to-video
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2 \
+  --pipeline-class-name LTX2Pipeline \
+  --prompt "A quiet coastal town at sunrise, fishing boats moving slowly through golden mist, cinematic camera movement" \
+  --save-output
+```
+
+#### 4.1.2 LTX-2.3 one-stage text-to-video
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2.3 \
+  --pipeline-class-name LTX2Pipeline \
+  --prompt "A quiet coastal town at sunrise, fishing boats moving slowly through golden mist, cinematic camera movement" \
+  --save-output
+```
+
+#### 4.1.3 LTX-2 two-stage text-to-video
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2 \
+  --pipeline-class-name LTX2TwoStagePipeline \
+  --prompt "A handheld shot follows a red tram crossing a rainy city square at night, reflections on the pavement, cinematic lighting" \
+  --save-output
+```
+
+#### 4.1.4 LTX-2.3 two-stage text-to-video
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2.3 \
+  --pipeline-class-name LTX2TwoStagePipeline \
+  --prompt "A handheld shot follows a red tram crossing a rainy city square at night, reflections on the pavement, cinematic lighting" \
+  --save-output
+```
+
+#### 4.1.5 LTX-2.3 HQ text-to-video
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2.3 \
+  --pipeline-class-name LTX2TwoStageHQPipeline \
+  --prompt "A wide cinematic shot of alpine clouds rolling over a mountain ridge, soft morning light, slow aerial camera movement" \
+  --save-output
+```
+
+#### 4.1.6 Image-to-video with one reference image
+
+Pass one image to `--image-path` for image-conditioned generation:
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2.3 \
+  --pipeline-class-name LTX2TwoStagePipeline \
+  --image-path ./inputs/start.png \
+  --prompt "The camera slowly pushes forward as the subject turns toward warm window light, subtle natural motion, cinematic" \
+  --save-output
+```
+
+#### 4.1.7 First-to-last-frame transition with two reference images
+
+Pass two images to `--image-path` for transition-style TI2V. The first image is used as the starting condition and the second image is used as the ending condition.
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2.3 \
+  --pipeline-class-name LTX2TwoStagePipeline \
+  --image-path ./inputs/start.png ./inputs/end.png \
+  --prompt "A smooth cinematic transition from the first scene into the final scene, dynamic camera motion, motion blur, zhuanchang" \
+  --save-output
+```
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Use community LoRAs
+
+Use `--lora-path` to load a LoRA adapter. If the Hugging Face repo contains multiple safetensors files, use `--lora-weight-name` to select the exact file. `--lora-scale` maps to the standard LoRA merge scale and defaults to `1.0`.
+
+The following example uses [`valiantcat/LTX-2.3-Transition-LORA`](https://huggingface.co/valiantcat/LTX-2.3-Transition-LORA):
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2.3 \
+  --pipeline-class-name LTX2TwoStagePipeline \
+  --lora-path valiantcat/LTX-2.3-Transition-LORA \
+  --lora-weight-name ltx2.3-transition.safetensors \
+  --prompt "A low-angle tracking shot moves through a foggy forest road. The camera rises above the treetops and transitions into a clear view of a snowy mountain peak under bright sunlight, zhuanchang" \
+  --save-output
+```
+
+You can combine the Transition LoRA with two reference images:
+
+```bash Command
+sglang generate \
+  --model-path Lightricks/LTX-2.3 \
+  --pipeline-class-name LTX2TwoStagePipeline \
+  --image-path ./inputs/start.png ./inputs/end.png \
+  --lora-path valiantcat/LTX-2.3-Transition-LORA \
+  --lora-weight-name ltx2.3-transition.safetensors \
+  --prompt "A fast cinematic transition from the first image to the second image, whip-pan motion, atmospheric lighting, zhuanchang" \
+  --save-output
+```
+
+<Note>
+Some community LoRAs only include weights for transformer blocks. In that case, SGLang logs a concise coverage summary and leaves unmatched LoRA-capable layers on the base model weights. This is expected when the adapter format intentionally omits those layers.
+</Note>
+
+## 5. Practical Tips
+
+- Use `--pipeline-class-name LTX2TwoStagePipeline` as the default LTX two-stage quality path.
+- Use `--pipeline-class-name LTX2TwoStageHQPipeline` when you want the HQ path and have enough VRAM for larger outputs.
+- Use `--ltx2-two-stage-device-mode resident` on high-VRAM GPUs if latency matters more than memory usage.
+- Use `--ltx2-two-stage-device-mode original` when comparing against official two-stage behavior.
+- Keep `--width` and `--height` aligned with the target model resolution; for LTX models, these are output video dimensions.
@@ -19,6 +19,12 @@ metatags:
    href="/cookbook/diffusion/Wan/Wan2.2"
    img="/cards/logos/wan.png"
  />
+  <Card
+    title="LTX"
+    mode="card"
+    href="/cookbook/diffusion/LTX/LTX"
+    img="/cards/Diffusion-card.png"
+  />
  <Card
    title="Qwen-Image"
    mode="card"
@@ -1094,6 +1094,12 @@
                      "cookbook/diffusion/Wan/Wan2.2"
                    ]
                  },
+                  {
+                    "group": "LTX",
+                    "pages": [
+                      "cookbook/diffusion/LTX/LTX"
+                    ]
+                  },
                  {
                    "group": "Qwen-Image",
                    "pages": [
@@ -0,0 +1,233 @@
+export const LTXDeployment = () => {
+  const options = {
+    hardware: {
+      name: 'hardware',
+      title: 'Hardware Platform',
+      items: [
+        { id: 'h200', label: 'H200', subtitle: 'Fastest, resident', default: true },
+        { id: 'standard', label: 'Standard CUDA', subtitle: 'Snapshot mode', default: false },
+        { id: 'official', label: 'Official Match', subtitle: 'Original switching', default: false },
+      ],
+    },
+    model: {
+      name: 'model',
+      title: 'Model',
+      items: [
+        { id: 'ltx23', label: 'LTX-2.3', default: true },
+        { id: 'ltx2', label: 'LTX-2', default: false },
+      ],
+    },
+    pipeline: {
+      name: 'pipeline',
+      title: 'Pipeline',
+      items: [
+        { id: 'two-stage', label: 'Two Stage', default: true, validModels: ['ltx2', 'ltx23'] },
+        { id: 'two-stage-hq', label: 'Two Stage HQ', subtitle: 'High Quality', default: false, validModels: ['ltx23'] },
+        { id: 'one-stage', label: 'One Stage', default: false, validModels: ['ltx2', 'ltx23'] },
+      ],
+    },
+  };
+
+  const modelConfigs = {
+    ltx2: {
+      repoId: 'Lightricks/LTX-2',
+      pipelines: {
+        'one-stage': 'LTX2Pipeline',
+        'two-stage': 'LTX2TwoStagePipeline',
+      },
+      supportedLoras: [],
+    },
+    ltx23: {
+      repoId: 'Lightricks/LTX-2.3',
+      pipelines: {
+        'one-stage': 'LTX2Pipeline',
+        'two-stage': 'LTX2TwoStagePipeline',
+        'two-stage-hq': 'LTX2TwoStageHQPipeline',
+      },
+      supportedLoras: [
+        {
+          id: 'transition',
+          path: 'valiantcat/LTX-2.3-Transition-LORA',
+          weightName: 'ltx2.3-transition.safetensors',
+          validPipelines: ['two-stage', 'two-stage-hq'],
+        },
+      ],
+    },
+  };
+
+  const getInitialState = () => ({
+    hardware: 'h200',
+    model: 'ltx23',
+    pipeline: 'two-stage',
+    selectedLoraPath: 'none',
+  });
+
+  const [values, setValues] = useState(getInitialState);
+  const [isDark, setIsDark] = useState(false);
+
+  useEffect(() => {
+    const checkDarkMode = () => {
+      const html = document.documentElement;
+      const isDarkMode = html.classList.contains('dark') ||
+        html.getAttribute('data-theme') === 'dark' ||
+        html.style.colorScheme === 'dark';
+      setIsDark(isDarkMode);
+    };
+    checkDarkMode();
+    const observer = new MutationObserver(checkDarkMode);
+    observer.observe(document.documentElement, { attributes: true, attributeFilter: ['class', 'data-theme', 'style'] });
+    return () => observer.disconnect();
+  }, []);
+
+  const availableLoras = (() => {
+    const config = modelConfigs[values.model];
+    return (config?.supportedLoras || []).filter((lora) => lora.validPipelines.includes(values.pipeline));
+  })();
+
+  const handleRadioChange = (optionName, itemId) => {
+    setValues((prev) => {
+      const next = { ...prev, [optionName]: itemId };
+
+      const validPipeline = options.pipeline.items.some((item) => (
+        item.id === next.pipeline && item.validModels.includes(next.model)
+      ));
+      if (!validPipeline) {
+        next.pipeline = 'two-stage';
+      }
+
+      const config = modelConfigs[next.model];
+      const nextSupported = (config?.supportedLoras || []).filter((lora) => lora.validPipelines.includes(next.pipeline));
+      const isValid = nextSupported.some((lora) => lora.path === prev.selectedLoraPath);
+      if (!isValid) {
+        next.selectedLoraPath = 'none';
+      }
+      return next;
+    });
+  };
+
+  const handleLoraToggle = (path) => {
+    setValues((prev) => ({
+      ...prev,
+      selectedLoraPath: prev.selectedLoraPath === path ? 'none' : path,
+    }));
+  };
+
+  const getDeviceMode = () => {
+    if (values.hardware === 'h200') {
+      return 'resident';
+    }
+    if (values.hardware === 'official') {
+      return 'original';
+    }
+    return 'snapshot';
+  };
+
+  const generateCommand = () => {
+    const config = modelConfigs[values.model];
+    const pipelineClass = config.pipelines[values.pipeline];
+    if (!pipelineClass) {
+      return '# Error: Invalid configuration';
+    }
+
+    let command = `sglang serve \\\n  --model-path ${config.repoId} \\\n  --pipeline-class-name ${pipelineClass}`;
+    if (values.pipeline !== 'one-stage') {
+      command += ` \\\n  --ltx2-two-stage-device-mode ${getDeviceMode()}`;
+    }
+
+    const selectedLora = availableLoras.find((lora) => lora.path === values.selectedLoraPath);
+    if (selectedLora) {
+      command += ` \\\n  --lora-path ${selectedLora.path} \\\n  --lora-weight-name ${selectedLora.weightName}`;
+    }
+
+    command += ` \\\n  --port 30000`;
+    return command;
+  };
+
+  const containerStyle = { maxWidth: '900px', margin: '0 auto', display: 'flex', flexDirection: 'column', gap: '4px' };
+  const cardStyle = { padding: '8px 12px', border: `1px solid ${isDark ? '#374151' : '#e5e7eb'}`, borderLeft: `3px solid ${isDark ? '#E85D4D' : '#D45D44'}`, borderRadius: '4px', display: 'flex', alignItems: 'center', gap: '12px', background: isDark ? '#1f2937' : '#fff' };
+  const titleStyle = { fontSize: '13px', fontWeight: '600', minWidth: '140px', flexShrink: 0, color: isDark ? '#e5e7eb' : 'inherit' };
+  const itemsStyle = { display: 'flex', rowGap: '2px', columnGap: '6px', flexWrap: 'wrap', alignItems: 'center', flex: 1 };
+  const labelBaseStyle = { padding: '4px 10px', border: `1px solid ${isDark ? '#9ca3af' : '#d1d5db'}`, borderRadius: '3px', cursor: 'pointer', display: 'inline-flex', flexDirection: 'column', alignItems: 'center', justifyContent: 'center', fontWeight: '500', fontSize: '13px', transition: 'all 0.2s', userSelect: 'none', minWidth: '45px', textAlign: 'center', flex: 1, background: isDark ? '#374151' : '#fff', color: isDark ? '#e5e7eb' : 'inherit' };
+  const checkedStyle = { background: '#D45D44', color: 'white', borderColor: '#D45D44' };
+  const subtitleStyle = { display: 'block', fontSize: '9px', marginTop: '1px', lineHeight: '1.1', opacity: 0.7 };
+  const commandDisplayStyle = { flex: 1, padding: '12px 16px', background: isDark ? '#111827' : '#f5f5f5', borderRadius: '6px', fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace", fontSize: '12px', lineHeight: '1.5', color: isDark ? '#e5e7eb' : '#374151', whiteSpace: 'pre-wrap', overflowX: 'auto', margin: 0, border: `1px solid ${isDark ? '#374151' : '#e5e7eb'}` };
+
+  return (
+    <div style={containerStyle} className="not-prose">
+      {Object.entries(options).map(([key, option]) => {
+        const itemsToDisplay = key === 'pipeline'
+          ? option.items.filter((item) => item.validModels.includes(values.model))
+          : option.items;
+
+        return (
+          <div key={key} style={cardStyle}>
+            <div style={titleStyle}>{option.title}</div>
+            <div style={itemsStyle}>
+              {itemsToDisplay.map((item) => {
+                const isChecked = values[option.name] === item.id;
+                return (
+                  <label key={item.id} style={{ ...labelBaseStyle, ...(isChecked ? checkedStyle : {}) }}>
+                    <input
+                      type="radio"
+                      name={option.name}
+                      checked={isChecked}
+                      onChange={() => handleRadioChange(key, item.id)}
+                      style={{ display: 'none' }}
+                    />
+                    {item.label}
+                    {item.subtitle && (
+                      <small style={{ ...subtitleStyle, color: isChecked ? 'rgba(255,255,255,0.85)' : 'inherit' }}>
+                        {item.subtitle}
+                      </small>
+                    )}
+                  </label>
+                );
+              })}
+            </div>
+          </div>
+        );
+      })}
+
+      <div style={cardStyle}>
+        <div style={titleStyle}>Select LoRA Model</div>
+        <div style={itemsStyle}>
+          {availableLoras.length === 0 && (
+            <div style={{ color: isDark ? '#999' : '#666', fontSize: '12px', padding: '8px' }}>
+              No LoRA models available for this configuration.
+            </div>
+          )}
+          {availableLoras.map((lora) => {
+            const isSelected = values.selectedLoraPath === lora.path;
+            return (
+              <label
+                key={lora.id}
+                style={{ ...labelBaseStyle, ...(isSelected ? checkedStyle : {}) }}
+                onClick={(event) => {
+                  event.preventDefault();
+                  handleLoraToggle(lora.path);
+                }}
+              >
+                <input
+                  type="radio"
+                  name="loraModelSelection"
+                  checked={isSelected}
+                  readOnly
+                  style={{ display: 'none' }}
+                />
+                {lora.id}
+                <small style={{ ...subtitleStyle, color: isSelected ? 'rgba(255,255,255,0.85)' : 'inherit' }}>
+                  {lora.path}
+                </small>
+              </label>
+            );
+          })}
+        </div>
+      </div>
+
+      <div style={cardStyle}>
+        <div style={titleStyle}>Run this Command:</div>
+        <pre style={commandDisplayStyle}>{generateCommand()}</pre>
+      </div>
+    </div>
+  );
+};