diff --git a/.github/workflows/nightly-test-nvidia.yml b/.github/workflows/nightly-test-nvidia.yml index 5ff529fd7..d8eeff7fe 100644 --- a/.github/workflows/nightly-test-nvidia.yml +++ b/.github/workflows/nightly-test-nvidia.yml @@ -202,7 +202,7 @@ jobs: - name: Checkout code uses: actions/checkout@v4 with: - ref: ${{ inputs.ref || github.ref }} + ref: ${{ inputs.ref || github.sha }} - name: Install dependencies run: | diff --git a/docs/cookbook/base/benchmarks/diffusion_model_benchmark.mdx b/docs/cookbook/base/benchmarks/diffusion_model_benchmark.mdx index 4c8e7a092..e4fdb6bc2 100644 --- a/docs/cookbook/base/benchmarks/diffusion_model_benchmark.mdx +++ b/docs/cookbook/base/benchmarks/diffusion_model_benchmark.mdx @@ -4,7 +4,7 @@ metatags: description: "Benchmark diffusion model serving throughput and latency with SGLang - supports image and video generation with flexible configurations." --- -`sglang.multimodal_gen.benchmarks.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of Diffusion Models. It supports two backends (`sglang-image`, `sglang-video`) and offers flexible configurations for request rates, dataset types, and profiling. +`sglang.multimodal_gen.benchmarks.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of diffusion models. It selects the image or video API from the requested task and offers flexible configurations for request rates, dataset types, and profiling. ## 1. Quick Start @@ -15,23 +15,23 @@ Run a benchmark on a local server (port 30000) generating 1 videos/images from t ```bash Command # For text to video: such as Wan2.2-T2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task t2v --num-prompts 1 --max-concurrency 1 + --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 # For image to video: such as Wan2.2-I2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task i2v --num-prompts 1 --max-concurrency 1 + --dataset vbench --task image-to-video --num-prompts 1 --max-concurrency 1 # For image-text to video: such as Wan2.2-TI2V-5B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task ti2v --num-prompts 1 --max-concurrency 1 + --dataset vbench --task image-to-video --num-prompts 1 --max-concurrency 1 # For text to image: such as Qwen-Image python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task t2i --num-prompts 1 --max-concurrency 1 + --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 # For image-text to image: such as Qwen-Image-Edit python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task ti2i --num-prompts 1 --max-concurrency 1 + --dataset vbench --task image-to-image --num-prompts 1 --max-concurrency 1 ``` ### 1.2 Benchmarking in High Concurrency @@ -41,28 +41,28 @@ Run a benchmark on a local server (port 30000) generating 20 videos/images from ```bash Command # For text to video: such as Wan2.2-T2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task t2v --num-prompts 20 --max-concurrency 20 + --dataset vbench --task text-to-video --num-prompts 20 --max-concurrency 20 # For image to video: such as Wan2.2-I2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task i2v --num-prompts 20 --max-concurrency 20 + --dataset vbench --task image-to-video --num-prompts 20 --max-concurrency 20 # For image-text to video: such as Wan2.2-TI2V-5B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task ti2v --num-prompts 20 --max-concurrency 20 + --dataset vbench --task image-to-video --num-prompts 20 --max-concurrency 20 # For text to image: such as Qwen-Image python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task t2i --num-prompts 20 --max-concurrency 20 + --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 # For image-text to image: such as Qwen-Image-Edit python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task ti2i --num-prompts 20 --max-concurrency 20 + --dataset vbench --task image-to-image --num-prompts 20 --max-concurrency 20 ``` ## 2. Parameter Reference -### 2.1 Connection & Backend Settings +### 2.1 Connection Settings @@ -78,11 +78,6 @@ python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - - - - - @@ -124,8 +119,8 @@ python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - - + + diff --git a/docs/cookbook/diffusion/FLUX/FLUX.mdx b/docs/cookbook/diffusion/FLUX/FLUX.mdx index a38345f6a..24bc85a0d 100644 --- a/docs/cookbook/diffusion/FLUX/FLUX.mdx +++ b/docs/cookbook/diffusion/FLUX/FLUX.mdx @@ -230,17 +230,16 @@ Test Environment: ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task t2v --num-prompts 1 --max-concurrency 1 + --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output ================= Serving Benchmark Result ================= - Backend: sglang-image Model: black-forest-labs/FLUX.1-dev Dataset: vbench - Task: t2v + Task: text-to-image -------------------------------------------------- Benchmark duration (s): 50.97 Request rate: inf @@ -317,17 +316,16 @@ Test Environment: ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task t2v --num-prompts 20 --max-concurrency 20 + --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result** : ```text Output ================= Serving Benchmark Result ================= - Backend: sglang-image Model: black-forest-labs/FLUX.1-dev Dataset: vbench - Task: t2v + Task: text-to-image -------------------------------------------------- Benchmark duration (s): 111.79 Request rate: inf diff --git a/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx b/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx index a9801bb8a..35b9bd804 100644 --- a/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx +++ b/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx @@ -67,8 +67,6 @@ For two-stage pipelines, `--ltx2-two-stage-device-mode` controls transformer res | `resident` | Best latency on high-VRAM GPUs because both DiTs can stay resident. | | `original` | Closest to the original two-stage switching semantics. | -`snapshot` is kept only as a deprecated compatibility alias for `original` and may be removed after two release cycles; use `original` or `resident` in new configs. - Other deployment flags: - `--lora-path`: Preload a community LoRA adapter. diff --git a/docs/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE.mdx b/docs/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE.mdx new file mode 100644 index 000000000..f34043b6a --- /dev/null +++ b/docs/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE.mdx @@ -0,0 +1,147 @@ +--- +title: LingBot Video MoE +description: Serve the native LingBot Video MoE 30B-A3B text-to-video model with SGLang Diffusion. +metatags: + description: "Run robbyant/lingbot-video-moe-30b-a3b text-to-video generation with SGLang Diffusion." +--- + +import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; + + + +## 1. Model introduction + +[LingBot Video MoE 30B-A3B](https://huggingface.co/robbyant/lingbot-video-moe-30b-a3b) +is a text-to-video mixture-of-experts model. SGLang Diffusion provides a native +pipeline for the public checkpoint: + +| Model ID | Task | Default output | +| --- | --- | --- | +| `robbyant/lingbot-video-moe-30b-a3b` | Text to video | 480x480, 81 frames at 16 FPS | + +The checkpoint expects a structured JSON caption rather than an unexpanded +natural-language prompt. The JSON is passed as the request's `prompt` string; +it is not an `extra_params` object. + +## 2. Installation + +Install SGLang with the diffusion dependencies: + +```bash Command +uv pip install "sglang[diffusion]" --prerelease=allow +``` + +See the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) +for platform-specific setup. + +## 3. Serve LingBot Video MoE + +Start the server with the Hugging Face model ID: + +```bash Command +sglang serve \ + --model-path robbyant/lingbot-video-moe-30b-a3b \ + --port 30010 +``` + +## 4. Generate a video + +The following request uses the compact 17-frame, 12-step smoke-test profile. +Use the model defaults of 81 frames and 40 steps for the released generation +profile. + +```python Python +import json +import time +from pathlib import Path + +import requests + +base_url = "http://127.0.0.1:30010" +prompt = json.dumps( + { + "comprehensive_description": { + "scene_content_description": ( + "A small silver robot arm on a white table slowly reaches " + "toward a red cube. The background is a softly lit laboratory wall." + ), + "camera_movement_description": ( + "The camera is static at eye level in a medium shot." + ), + }, + "camera_info": { + "color": "Neutral", + "frame_size": "Medium", + "shot_type_angle": "Eye level", + "lens_size": "Medium", + "composition": "Center", + "lighting": "Soft light", + "lighting_type": "Artificial light", + }, + "world_knowledge": [], + "prominent_elements": [ + { + "name": "robot arm", + "description": "A small silver robot arm with a two-finger gripper.", + "actions": [ + { + "timestamp": "[0.0s - 1.0s]", + "action": "reaches toward the red cube", + } + ], + "location": "center of the frame", + "relative_size": "dominant", + "shape_and_color": "articulated silver metal arm", + "texture": "brushed metal", + "appearance_details": "two-finger gripper and visible joints", + "relationship": "reaching toward the red cube on the table", + "orientation": "upright, base on the table", + "pose": "reaching", + } + ], + }, + separators=(",", ":"), +) + +response = requests.post( + f"{base_url}/v1/videos", + json={ + "model": "robbyant/lingbot-video-moe-30b-a3b", + "prompt": prompt, + "size": "640x384", + "num_frames": 17, + "fps": 16, + "num_inference_steps": 12, + "guidance_scale": 6.0, + "flow_shift": 3.0, + "seed": 0, + }, + timeout=60, +) +response.raise_for_status() +video_id = response.json()["id"] + +while True: + job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json() + if job["status"] == "completed": + break + if job["status"] == "failed": + raise RuntimeError(job.get("error") or "Video generation failed") + time.sleep(1) + +video = requests.get( + f"{base_url}/v1/videos/{video_id}/content", + timeout=300, +) +video.raise_for_status() +Path("lingbot_video_moe.mp4").write_bytes(video.content) +``` + +## 5. Request constraints + +- `num_frames` must be `1` or `4n+1`; examples include 17 and 81. +- Width and height must both be multiples of 16. +- The native defaults are `guidance_scale=6.0`, `flow_shift=3.0`, + `num_inference_steps=40`, and `fps=16`. +- Keep the prompt as serialized JSON. Raw free text is outside the + checkpoint's expected caption format. diff --git a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx index fc29c905e..f25dbebd6 100644 --- a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx +++ b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx @@ -220,17 +220,16 @@ sglang serve --model-path Qwen/Qwen-Image-Edit-2511 --port 30000 ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task ti2i --num-prompts 1 --max-concurrency 1 + --dataset vbench --task image-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output ================= Serving Benchmark Result ================= -Backend: sglang-image Model: Qwen/Qwen-Image-Edit-2511 Dataset: vbench -Task: ti2i +Task: image-to-image -------------------------------------------------- Benchmark duration (s): 35.31 Request rate: inf @@ -254,17 +253,16 @@ Peak Memory Median (MB): 47959.35 ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task ti2i --num-prompts 20 --max-concurrency 20 + --dataset vbench --task image-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output ================= Serving Benchmark Result ================= -Backend: sglang-image Model: Qwen/Qwen-Image-Edit-2511 Dataset: vbench -Task: ti2i +Task: image-to-image -------------------------------------------------- Benchmark duration (s): 286.11 Request rate: inf diff --git a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx index 8f1250e47..ba657ed11 100644 --- a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx +++ b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx @@ -237,7 +237,7 @@ Test Environment: ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 + --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: @@ -317,7 +317,7 @@ Test Environment: ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 --port 30000 + --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 --port 30000 ``` **Result**: diff --git a/docs/cookbook/diffusion/SANA-Video/SANA-Video.mdx b/docs/cookbook/diffusion/SANA-Video/SANA-Video.mdx new file mode 100644 index 000000000..ad3c67ab6 --- /dev/null +++ b/docs/cookbook/diffusion/SANA-Video/SANA-Video.mdx @@ -0,0 +1,96 @@ +--- +title: SANA-Video +description: Serve the native SANA-Video 2B 480p text-to-video model with SGLang Diffusion. +metatags: + description: "Run Efficient-Large-Model/SANA-Video_2B_480p_diffusers text-to-video generation with SGLang Diffusion." +--- + +import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; + + + +## 1. Model introduction + +[SANA-Video 2B 480p](https://huggingface.co/Efficient-Large-Model/SANA-Video_2B_480p_diffusers) +is a text-to-video model with a native SGLang Diffusion pipeline. + +| Model ID | Task | Default output | +| --- | --- | --- | +| `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | Text to video | 832x480, 81 frames at 16 FPS | + +## 2. Installation + +Install SGLang with the diffusion dependencies: + +```bash Command +uv pip install "sglang[diffusion]" --prerelease=allow +``` + +See the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) +for platform-specific setup. + +## 3. Serve SANA-Video + +```bash Command +sglang serve \ + --model-path Efficient-Large-Model/SANA-Video_2B_480p_diffusers \ + --port 30010 +``` + +## 4. Generate a video + +The following request uses the compact 17-frame, 8-step profile covered by +server CI. Use the model defaults of 81 frames and 50 steps for the released +generation profile. + +```python Python +import time +from pathlib import Path + +import requests + +base_url = "http://127.0.0.1:30010" +response = requests.post( + f"{base_url}/v1/videos", + json={ + "model": "Efficient-Large-Model/SANA-Video_2B_480p_diffusers", + "prompt": ( + "A red tram moves slowly through a sunlit city square while " + "pedestrians cross behind it. motion score: 30." + ), + "size": "832x480", + "num_frames": 17, + "fps": 16, + "num_inference_steps": 8, + "guidance_scale": 6.0, + "seed": 42, + }, + timeout=60, +) +response.raise_for_status() +video_id = response.json()["id"] + +while True: + job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json() + if job["status"] == "completed": + break + if job["status"] == "failed": + raise RuntimeError(job.get("error") or "Video generation failed") + time.sleep(1) + +video = requests.get( + f"{base_url}/v1/videos/{video_id}/content", + timeout=300, +) +video.raise_for_status() +Path("sana_video.mp4").write_bytes(video.content) +``` + +## 5. Request constraints + +- The default profile uses `832x480`, 81 frames, 50 inference steps, and 16 FPS. +- Frame counts are aligned to `4n+1`; for example, a request for 80 frames is + adjusted to 77. +- Use width and height values divisible by 16. +- The prompt supports an optional `motion score: N.` suffix to express the + desired amount of motion. diff --git a/docs/cookbook/diffusion/Wan/Wan2.1.mdx b/docs/cookbook/diffusion/Wan/Wan2.1.mdx index 7ee3df0e8..de9b21aef 100644 --- a/docs/cookbook/diffusion/Wan/Wan2.1.mdx +++ b/docs/cookbook/diffusion/Wan/Wan2.1.mdx @@ -200,7 +200,7 @@ You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 pe ```bash Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 + --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: @@ -304,7 +304,7 @@ You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 pe ```bash Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 + --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: diff --git a/docs/cookbook/diffusion/Wan/Wan2.2.mdx b/docs/cookbook/diffusion/Wan/Wan2.2.mdx index fc5daab12..6e666869a 100644 --- a/docs/cookbook/diffusion/Wan/Wan2.2.mdx +++ b/docs/cookbook/diffusion/Wan/Wan2.2.mdx @@ -270,16 +270,15 @@ Test Environment: **Benchmark Command**: ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task t2v --num-prompts 1 --max-concurrency 1 + --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output ================= Serving Benchmark Result ================= - Backend: sglang-video Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers Dataset: vbench - Task: t2v + Task: text-to-video -------------------------------------------------- Benchmark duration (s): 630.43 Request rate: inf @@ -372,17 +371,16 @@ Test Environment: ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-video --dataset vbench --task t2v --num-prompts 20 --max-concurrency 20 + --dataset vbench --task text-to-video --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output ================= Serving Benchmark Result ================= - Backend: sglang-video Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers Dataset: vbench - Task: t2v + Task: text-to-video -------------------------------------------------- Benchmark duration (s): 5163.21 Request rate: inf diff --git a/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx b/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx index e33114dea..255406a0e 100644 --- a/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx +++ b/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx @@ -225,7 +225,7 @@ Test Environment: ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 + --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: @@ -305,7 +305,7 @@ Test Environment: ```shell Command python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ - --backend sglang-image --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 + --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: diff --git a/docs/cookbook/diffusion/intro.mdx b/docs/cookbook/diffusion/intro.mdx index c9cf90010..5ecc49011 100644 --- a/docs/cookbook/diffusion/intro.mdx +++ b/docs/cookbook/diffusion/intro.mdx @@ -80,6 +80,12 @@ Video models denoise a bounded latent video sequence for each request. Use these href="/cookbook/diffusion/LTX/LTX2 & LTX2.3" img="/cards/logos/ltx.svg" /> + + ## Realtime / World Models diff --git a/docs/docs.json b/docs/docs.json index 20c5c537e..b06207115 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1512,6 +1512,13 @@ "cookbook/diffusion/MOVA/MOVA" ] }, + { + "group": "SANA Video", + "tag": "NEW", + "pages": [ + "cookbook/diffusion/SANA-Video/SANA-Video" + ] + }, { "group": "MiniMax", "tag": "NEW", @@ -1519,6 +1526,13 @@ "cookbook/diffusion/MiniMax/MiniMax-H3" ] }, + { + "group": "LingBot Video", + "tag": "NEW", + "pages": [ + "cookbook/diffusion/LingBot-Video/LingBot-Video-MoE" + ] + }, { "group": "LingBot World", "pages": [ diff --git a/docs/docs/sglang-diffusion/compatibility_matrix.mdx b/docs/docs/sglang-diffusion/compatibility_matrix.mdx index e13a09473..f07ef109e 100644 --- a/docs/docs/sglang-diffusion/compatibility_matrix.mdx +++ b/docs/docs/sglang-diffusion/compatibility_matrix.mdx @@ -89,6 +89,12 @@ Rows are grouped when a family shares the same runtime path or optimization supp + + + + + + @@ -285,6 +291,21 @@ Optimization columns are abbreviated to keep the matrix readable: + + + + + + + + + + + + + + + @@ -678,7 +699,6 @@ Optimization columns are abbreviated to keep the matrix readable: - `original` keeps official two-stage semantics without the premerged stage-2 transformer path. - `resident` usually provides the best latency/throughput but uses much more VRAM. - Default is auto: `resident` on H200/high-memory CUDA GPUs, otherwise `original`. - - Deprecated compatibility: `snapshot` is accepted as an alias for `original` and may be removed after two release cycles. 5. Cosmos3 ships in two sizes — `nvidia/Cosmos3-Nano` (16B) and `nvidia/Cosmos3-Super` (64B). Both share the same pipeline; the only difference is transformer depth and width, picked up from diff --git a/python/sglang/multimodal_gen/README.md b/python/sglang/multimodal_gen/README.md index 8602dfa05..05ad5cf35 100644 --- a/python/sglang/multimodal_gen/README.md +++ b/python/sglang/multimodal_gen/README.md @@ -9,7 +9,7 @@ SGLang diffusion features an end-to-end unified pipeline for accelerating diffus ## Key Features SGLang Diffusion has the following features: - - Broad model support: Wan, FastWan, FLUX, Qwen-Image, Z-Image, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3, MiniMax-H3, LingBot World, SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more + - Broad model support: Wan, FastWan, FLUX, Qwen-Image, Z-Image, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3, MiniMax-H3, LingBot Video MoE, LingBot World, SANA-Video/SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more - Fast inference speed: empowered by optimized `sgl-kernel` kernels, scheduler/runtime improvements, caching acceleration, and native diffusion hot-path optimizations - Ease of use: OpenAI-compatible api, CLI, and python sdk support - Multi-platform support: diff --git a/python/sglang/multimodal_gen/benchmarks/bench_serving.py b/python/sglang/multimodal_gen/benchmarks/bench_serving.py index d50f98bdf..dca9a4c28 100644 --- a/python/sglang/multimodal_gen/benchmarks/bench_serving.py +++ b/python/sglang/multimodal_gen/benchmarks/bench_serving.py @@ -728,12 +728,6 @@ if __name__ == "__main__": parser = argparse.ArgumentParser( description="Benchmark serving for diffusion models." ) - parser.add_argument( - "--backend", - type=str, - default=None, - help="DEPRECATED: --task is deprecated and will be ignored. The task will be inferred from --model.", - ) parser.add_argument( "--base-url", type=str, diff --git a/python/sglang/multimodal_gen/runtime/entrypoints/utils.py b/python/sglang/multimodal_gen/runtime/entrypoints/utils.py index 3eee51ed1..e964b91a1 100644 --- a/python/sglang/multimodal_gen/runtime/entrypoints/utils.py +++ b/python/sglang/multimodal_gen/runtime/entrypoints/utils.py @@ -639,11 +639,12 @@ def _try_save_cuda_video_direct( stderr=stderr_file.read(), ) return True - except Exception as e: - logger.warning( - "Direct CUDA video save failed; falling back to imageio: %s", - str(e), + except Exception: + logger.warning_once( + "Direct CUDA video save failed; falling back to imageio. " + "Enable debug logging for exception details." ) + logger.debug("Direct CUDA video save failure", exc_info=True) return False finally: if tmp_wav_path: @@ -725,11 +726,12 @@ def _try_save_cuda_videos_direct( try: with ThreadPoolExecutor(max_workers=_MAX_PARALLEL_CUDA_VIDEO_SAVES) as pool: return list(pool.map(save_one, range(len(samples)))) - except Exception as exc: - logger.warning( - "Parallel CUDA video save failed; falling back to serial output: %s", - str(exc), + except Exception: + logger.warning_once( + "Parallel CUDA video save failed; falling back to serial output. " + "Enable debug logging for exception details." ) + logger.debug("Parallel CUDA video save failure", exc_info=True) return None diff --git a/python/sglang/multimodal_gen/runtime/platforms/cuda.py b/python/sglang/multimodal_gen/runtime/platforms/cuda.py index 5b89fb765..1c4bc869f 100644 --- a/python/sglang/multimodal_gen/runtime/platforms/cuda.py +++ b/python/sglang/multimodal_gen/runtime/platforms/cuda.py @@ -151,8 +151,7 @@ class _SageAttentionBackendResolver(_CudaAttentionBackendResolver): def resolve(cls, platform) -> str | AttentionBackendEnum: try: from sageattention import sageattn # noqa: F401 - except ImportError as e: - logger.info(e) + except ImportError: logger.info( "Sage Attention backend is not installed (To install it, run `pip install git+https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4 --no-build-isolation`). Falling back to Flash Attention." ) @@ -176,8 +175,7 @@ class _SageAttentionBackendResolver(_CudaAttentionBackendResolver): ) return "sglang.multimodal_gen.runtime.layers.attention.backends.sage_attn.SageAttentionBackend" - except ImportError as e: - logger.info(e) + except ImportError: logger.info( "Sage Attention backend failed to import. Falling back to Flash Attention." ) @@ -195,8 +193,7 @@ class _SageAttention3BackendResolver(_CudaAttentionBackendResolver): ) return "sglang.multimodal_gen.runtime.layers.attention.backends.sage_attn3.SageAttention3Backend" - except ImportError as e: - logger.info(e) + except ImportError: logger.info( "Sage Attention 3 backend is not installed (To install it, see https://github.com/thu-ml/SageAttention/tree/main/sageattention3_blackwell#installation). Falling back to Torch SDPA." ) diff --git a/python/sglang/multimodal_gen/runtime/server_args/server_args.py b/python/sglang/multimodal_gen/runtime/server_args/server_args.py index 8fa6e1f35..043572260 100644 --- a/python/sglang/multimodal_gen/runtime/server_args/server_args.py +++ b/python/sglang/multimodal_gen/runtime/server_args/server_args.py @@ -86,7 +86,7 @@ from sglang.multimodal_gen.utils import ( logger = init_logger(__name__) LTX2_TWO_STAGE_DEVICE_MODES = ("original", "resident") -LTX2_TWO_STAGE_DEVICE_MODE_CHOICES = (*LTX2_TWO_STAGE_DEVICE_MODES, "snapshot") +LTX2_TWO_STAGE_DEVICE_MODE_CHOICES = LTX2_TWO_STAGE_DEVICE_MODES LTX2_TWO_STAGE_PIPELINE_NAMES = ("LTX2TwoStagePipeline", "LTX2TwoStageHQPipeline") # H200-class GPUs (>=130 GiB total) can usually keep both LTX2 DiTs resident. LTX2_RESIDENT_AUTO_ENABLE_MEM_GB = 130 @@ -100,15 +100,7 @@ RING_CAPABLE_ATTENTION_BACKENDS = ("fa", "sage_attn") def _normalize_ltx2_two_stage_device_mode(mode: str | None) -> str | None: if mode is None: return None - mode = mode.lower() - if mode == "snapshot": - logger.warning( - "ltx2_two_stage_device_mode=snapshot is deprecated and is treated " - "as original. Please use ltx2_two_stage_device_mode=original or " - "resident instead. This alias may be removed after two release cycles." - ) - return "original" - return mode + return mode.lower() def is_ltx2_two_stage_pipeline_name(pipeline_class_name: str | None) -> bool: @@ -2211,8 +2203,6 @@ class ServerArgs(DisaggServerArgsMixin): "LTX-2.3 two-stage device residency mode: " "'original' keeps official two-stage semantics without premerged stage2, " "'resident' keeps both transformers resident on GPU. " - "'snapshot' is deprecated, treated as 'original', and may be " - "removed after two release cycles. " "Default is auto: resident on H200/high-memory CUDA GPUs, otherwise original." ), ) diff --git a/python/sglang/multimodal_gen/test/unit/test_server_args.py b/python/sglang/multimodal_gen/test/unit/test_server_args.py index 659cb0734..8a5ec7804 100644 --- a/python/sglang/multimodal_gen/test/unit/test_server_args.py +++ b/python/sglang/multimodal_gen/test/unit/test_server_args.py @@ -2001,24 +2001,19 @@ class TestOffloadDefaults(unittest.TestCase): ["text_encoder", "image_encoder", "vae"], ) - def test_ltx23_snapshot_device_mode_is_deprecated_alias_for_original(self): - args = self._from_dict_with_pipeline_config( - LTX2PipelineConfig(), - memory_gb=140, - available_memory_gb=134, - kwargs={ - "model_path": "Lightricks/LTX-2.3", - "num_gpus": 2, - "pipeline_class_name": "LTX2TwoStagePipeline", - "ltx2_two_stage_device_mode": "snapshot", - }, - ) - - self.assertEqual(args.ltx2_two_stage_device_mode, "original") - self.assertEqual( - args.layerwise_offload_components, - ["text_encoder", "image_encoder", "vae"], - ) + def test_ltx23_snapshot_device_mode_is_rejected(self): + with self.assertRaisesRegex(ValueError, "Expected one of"): + self._from_dict_with_pipeline_config( + LTX2PipelineConfig(), + memory_gb=140, + available_memory_gb=134, + kwargs={ + "model_path": "Lightricks/LTX-2.3", + "num_gpus": 2, + "pipeline_class_name": "LTX2TwoStagePipeline", + "ltx2_two_stage_device_mode": "snapshot", + }, + ) def test_explicit_layerwise_components_preserved_in_ltx23_resident(self): args = self._from_dict_with_pipeline_config( @@ -2425,7 +2420,7 @@ class TestOffloadDefaults(unittest.TestCase): self.assertFalse(server_args.use_fsdp_inference) self.assertFalse(server_args.enable_cfg_parallel) - def test_ltx23_snapshot_device_mode_cli_alias_is_accepted(self): + def test_ltx23_snapshot_device_mode_cli_is_rejected(self): parser = FlexibleArgumentParser() ServerArgs.add_cli_args(parser) argv = [ @@ -2437,36 +2432,8 @@ class TestOffloadDefaults(unittest.TestCase): "snapshot", ] - with ( - patch.object(sys, "argv", ["sglang"] + argv), - patch.object( - PipelineConfig, "from_kwargs", return_value=LTX2PipelineConfig() - ), - patch( - "sglang.multimodal_gen.runtime.platforms.current_platform.is_cpu", - return_value=False, - ), - patch( - "sglang.multimodal_gen.runtime.platforms.current_platform.is_mps", - return_value=False, - ), - patch( - "sglang.multimodal_gen.runtime.platforms.current_platform.is_cuda", - return_value=True, - ), - patch( - "sglang.multimodal_gen.runtime.platforms.current_platform.get_device_total_memory", - return_value=140 * 1024**3, - ), - patch( - "sglang.multimodal_gen.runtime.platforms.current_platform.get_available_gpu_memory", - return_value=134, - ), - ): - args, unknown_args = parser.parse_known_args(argv) - server_args = ServerArgs.from_cli_args(args, unknown_args) - - self.assertEqual(server_args.ltx2_two_stage_device_mode, "original") + with self.assertRaises(SystemExit): + parser.parse_known_args(argv) class TestKVGatherDegree(unittest.TestCase): diff --git a/scripts/ci/utils/diffusion/run_comparison.py b/scripts/ci/utils/diffusion/run_comparison.py index 62c64c963..86d765130 100644 --- a/scripts/ci/utils/diffusion/run_comparison.py +++ b/scripts/ci/utils/diffusion/run_comparison.py @@ -868,6 +868,20 @@ def _install_framework(fw_name: str, dry_run: bool = False) -> bool: return True +def _get_checkout_commit_sha() -> str: + fallback = os.environ.get("GITHUB_SHA", "unknown") + try: + result = subprocess.run( + ["git", "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ) + except (OSError, subprocess.CalledProcessError): + return fallback + return result.stdout.strip() or fallback + + def run_comparison( config: dict, case_ids: list[str] | None = None, @@ -882,7 +896,7 @@ def run_comparison( Each non-sglang framework is installed right before its cases run. """ timestamp = datetime.now(timezone.utc).isoformat() - commit_sha = os.environ.get("GITHUB_SHA", "unknown") + commit_sha = _get_checkout_commit_sha() run_id = os.environ.get("GITHUB_RUN_ID", "local") log_dir = Path("comparison-logs")
`--backend`**Required**The backend type to use. Choices: `sglang-image`, `sglang-video`.
`--base-url` `None`
`--task``t2v`, `i2v`, `ti2v`, `t2i`, `ti2i`Defines the generation task: `t2v` (Text-to-Video), `i2v` (Image-to-Video), `ti2v` (Text+Image-to-Video), `t2i` (Text-to-image), `ti2i` (Text+Image-to-Image).`text-to-video`, `image-to-video`, `text-to-image`, `image-to-image`, `video-to-video`Defines the generation task when it cannot be inferred from the model metadata.
`--dataset`T2V, 480p No dedicated optimization listed
LingBot Video MoE
robbyant/lingbot-video-moe-30b-a3b
T2V, 480pNo dedicated optimization listed
Wan2.2
Wan-AI/Wan2.2-TI2V-5B-DiffusersWan-AI/Wan2.2-T2V-A14B-Diffusersnvidia/Wan2.2-T2V-A14B-Diffusers-NVFP4Wan-AI/Wan2.2-I2V-A14B-Diffusers
❌ ❌
LingBot Video MoE 30B-A3Brobbyant/lingbot-video-moe-30b-a3b480p❌❌❌❌❌❌❌❌❌❌
FastWan2.2 TI2V 5B FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers
FastVideo/FastWan2.2-TI2V-5B-Diffusers