minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[diffusion] doc: update cosmos3 cookbook (#27576)

Browse Source 
Co-authored-by: Mick <mickjagger19@icloud.com>
This commit is contained in:
Richard Wang
2026-07-10 11:19:52 +08:00
committed by GitHub
co-authored by Mick
parent 7966f6be86
commit a38cfc6768
1 changed files with 77 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/diffusion/Cosmos/Cosmos3.mdx
+77 -10
View File
@@ -1,22 +1,22 @@
---
title: Cosmos3
metatags:
description: "Serve NVIDIA Cosmos3 text-to-video, image-to-video, and text-to-image generation with SGLang Diffusion."
description: "Serve NVIDIA Cosmos3 image, video, sound, and action generation with SGLang Diffusion."
---
## 1. Model Introduction
[NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is a world-generation model family for text-to-image, text-to-video, and image-to-video generation. SGLang Diffusion serves the public generator checkpoints with the native `Cosmos3OmniDiffusersPipeline`.
[NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is an omnimodal world-model family for image, video, sound, and action generation. SGLang Diffusion serves the public checkpoints with the native `Cosmos3OmniDiffusersPipeline`.
| Model | Status | Notes |
| --- | --- | --- |
| `nvidia/Cosmos3-Nano` | Supported | T2I, T2V, I2V |
| `nvidia/Cosmos3-Super` | Supported | T2I, T2V, I2V; use multi-GPU for the 64B checkpoint |
| `nvidia/Cosmos3-Nano` | Supported | T2I, T2V, I2V, V2V, joint sound, and action |
| `nvidia/Cosmos3-Super` | Supported | T2I, T2V, I2V, and V2V; use multi-GPU for the 32B checkpoint |
| `nvidia/Cosmos3-Super-Text2Image` | Supported | T2I-specialized checkpoint |
| `nvidia/Cosmos3-Super-Image2Video` | Supported | I2V-specialized checkpoint |
| `nvidia/Cosmos3-Nano-Policy-DROID` | Not supported yet | Action/policy model; planned separately from visual generation |
| `nvidia/Cosmos3-Nano-Policy-DROID` | Supported | DROID policy action generation |
Cosmos3 video-with-sound, video-to-video conditioning, and action generation are not supported yet. Requests that set `generate_sound`, `action_mode`, or video-to-video conditioning fields return a clear error instead of being silently ignored.
Sound and action generation require the corresponding checkpoint heads. SGLang uses the flow-native `FlowUniPCMultistepScheduler` for Cosmos3 even if the checkpoint metadata names another scheduler. The default `flow_shift` is `3.0` for T2I and `10.0` for video and action modes.
## 2. Installation
@@ -69,7 +69,7 @@ curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
"n": 1,
"num_inference_steps": 35,
"guidance_scale": 6.0,
"flow_shift": 10.0,
"flow_shift": 3.0,
"seed": 0,
"extra_args": {
"use_resolution_template": false,
@@ -78,9 +78,9 @@ curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
}'
```
### Text to video
### Text to video with sound
Use `/v1/videos` to create an asynchronous job, then poll the job and download the completed MP4.
Use `/v1/videos` to create an asynchronous job, then poll the job and download the completed MP4. Set `generate_sound=true` to generate and mux a stereo 48 kHz audio track; omit it for a silent video.
```bash Command
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
@@ -92,6 +92,7 @@ job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
--form-string "num_inference_steps=35" \
--form-string "guidance_scale=4.0" \
--form-string "flow_shift=10.0" \
--form-string "generate_sound=true" \
--form-string "seed=42" \
--form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
| python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
@@ -172,6 +173,58 @@ response.raise_for_status()
Path("cosmos3_i2v.mp4").write_bytes(response.content)
```
### Video to video
Upload a source video with `video_reference`. Cosmos3 keeps latent frames `[0, 1]` by default and generates the remaining frames. Use `condition_frame_indexes` to select different latent frames, and `condition_video_keep` to take conditioning frames from the start or end of the source.
```bash Command
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
--form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \
--form "video_reference=@robot_pouring.mp4;type=video/mp4" \
--form-string "size=1280x704" \
--form-string "num_frames=45" \
--form-string "fps=24" \
--form-string "num_inference_steps=35" \
--form-string "guidance_scale=6.0" \
--form-string 'condition_frame_indexes=[0,1]' \
--form-string "condition_video_keep=first" \
| python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
```
Poll and download this job with the same status and content endpoints used by the T2V example.
### Action generation
For DROID policy generation, start a single-GPU server with the policy checkpoint. Cosmos3 action generation does not currently support CFG or sequence parallelism.
```bash Command
sglang serve \
--model-path nvidia/Cosmos3-Nano-Policy-DROID \
--num-gpus 1
```
The following request predicts a 16-step action chunk from one observation. The chunk length is `num_frames - 1`, and the completed job's `action` field contains the tensor data, shape, mode, and active action dimension.
```bash Command
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
--form-string "prompt=Put the pot to the left of the purple item." \
--form "input_reference=@observation.png;type=image/png" \
--form-string "size=832x480" \
--form-string "num_frames=17" \
--form-string "fps=5" \
--form-string "num_inference_steps=30" \
--form-string "guidance_scale=1.0" \
--form-string "action_mode=policy" \
--form-string "domain_name=droid_lerobot" \
| python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
# After the job reaches "completed":
curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \
| python -c 'import json, sys; print(json.dumps(json.load(sys.stdin)["action"], indent=2))'
```
The other action modes are `forward_dynamics` (condition on an observation and an `action` JSON array to generate video) and `inverse_dynamics` (condition on a full video to predict action). Select the embodiment head with `domain_name` or `domain_id`; set `raw_action_dim` explicitly when it cannot be inferred from the domain name.
## 5. Cosmos3 Parameters
Cosmos3 supports the standard SGLang video and image fields such as `size`, `num_frames`, `fps`, `num_inference_steps`, `guidance_scale`, `negative_prompt`, and `seed`.
@@ -179,7 +232,21 @@ Cosmos3 supports the standard SGLang video and image fields such as `size`, `num
Top-level Cosmos3 request fields:
- `max_sequence_length`: maximum text token length used by the Cosmos3 tokenizer.
- `flow_shift`: per-request scheduler flow shift. If omitted, SGLang uses `--flow-shift`, then the checkpoint scheduler default.
- `flow_shift`: per-request scheduler shift. If omitted, SGLang uses `--flow-shift`, then the mode default (`3.0` for T2I and `10.0` for video/action).
Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields:
- `generate_sound`: generate a sound track whose duration follows `num_frames / fps`.
- `sound_duration`: explicit sound duration in seconds; takes precedence over the derived duration.
- `condition_frame_indexes`: V2V latent-frame indexes to keep from the source video; defaults to `[0, 1]`.
- `condition_video_keep`: use the `first` or `last` source frames for V2V conditioning.
- `action_mode`: `policy`, `forward_dynamics`, or `inverse_dynamics`.
- `domain_name` / `domain_id`: select the action embodiment head.
- `raw_action_dim`: number of active action dimensions; inferred for known domain names.
- `action`: action array with shape `[T, D]`, required by `forward_dynamics`.
- `action_fps`: action-token frame rate for temporal mRoPE; defaults to the video FPS.
- `action_view_point`: viewpoint used in the structured action caption.
- `action_normalization`: dataset normalization mode, such as `quantile`, `meanstd`, or `minmax`.
Put model-specific compatibility knobs in `extra_params` for video requests, or `extra_args` for image requests: