[Docs] Sync docs_new with legacy docs and update migration redirects (#23337)
Co-authored-by: Mingyi <wisclmy0611@gmail.com>
This commit is contained in:
zijiexia
@@ -3,295 +3,255 @@ title: CLI reference
|
||||
sidebarTitle: CLI
|
||||
description: Run one-off generation tasks and launch the HTTP server from the command line.
|
||||
---
|
||||
Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`.
|
||||
|
||||
The `sglang` CLI provides two main subcommands for diffusion inference:
|
||||
### Overlay repos for non-diffusers models
|
||||
|
||||
- **`sglang generate`** -- run a one-off generation without a persistent server
|
||||
- **`sglang serve`** -- launch the OpenAI-compatible HTTP server
|
||||
If `--model-path` points to a supported non-diffusers source repo, SGLang can resolve it
|
||||
through a self-hosted overlay repo.
|
||||
|
||||
## Prerequisites
|
||||
SGLang first checks a built-in overlay registry. Concrete built-in mappings can be added over time without changing the CLI surface.
|
||||
|
||||
A working SGLang Diffusion installation with the `sglang` CLI available in your `$PATH`. See the [installation guide](../installation) for setup instructions.
|
||||
Override example:
|
||||
|
||||
```bash Command
|
||||
export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{
|
||||
"Wan-AI/Wan2.2-S2V-14B": {
|
||||
"overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay",
|
||||
"overlay_revision": "main"
|
||||
}
|
||||
}'
|
||||
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-S2V-14B \
|
||||
--config configs/wan_s2v.yaml
|
||||
```
|
||||
|
||||
The overlay repo should be a complete diffusers-style/componentized repo
|
||||
|
||||
You can also pass the overlay repo itself as `--model-path` if it contains `_overlay/overlay_manifest.json`.
|
||||
|
||||
Notes:
|
||||
1. `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is only an optional override for
|
||||
development and debugging. It accepts either a JSON object or a path to a JSON
|
||||
file, and can extend or replace built-in entries for the current process.
|
||||
2. On the first load, SGLang will:
|
||||
- download overlay metadata from the overlay repo
|
||||
- download the required files from the original source repo
|
||||
- materialize a local standard component repo under `~/.cache/sgl_diffusion/materialized_models/`
|
||||
3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory.
|
||||
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Generate
|
||||
|
||||
```bash Command
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
### Serve
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--num-gpus 4 \
|
||||
--ulysses-degree 2 \
|
||||
--ring-degree 2 \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
For request and response examples, see [OpenAI-Compatible API](./openai_api).
|
||||
|
||||
<Tip>
|
||||
Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags.
|
||||
</Tip>
|
||||
|
||||
## Common Options
|
||||
|
||||
### Model and runtime
|
||||
|
||||
- `--model-path {MODEL}`: model path or Hugging Face model ID
|
||||
- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
|
||||
- `--num-gpus {N}`: number of GPUs to use
|
||||
- `--tp-size {N}`: tensor parallelism size, mainly for encoders
|
||||
- `--sp-degree {N}`: sequence parallelism size
|
||||
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
|
||||
- `--attention-backend {BACKEND}`: attention backend for native SGLang pipelines
|
||||
- `--attention-backend-config {CONFIG}`: attention backend configuration
|
||||
|
||||
### Sampling and output
|
||||
|
||||
- `--prompt {PROMPT}` and `--negative-prompt {PROMPT}`
|
||||
- `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation
|
||||
- `--num-inference-steps {STEPS}` and `--seed {SEED}`
|
||||
- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
|
||||
- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`
|
||||
|
||||
For frame interpolation and upscaling, see [Post-Processing](./post_processing).
|
||||
|
||||
### Quantized transformers
|
||||
|
||||
For quantized transformer checkpoints, prefer:
|
||||
|
||||
- `--model-path` for the base pipeline
|
||||
- `--transformer-path` for a quantized `transformers` transformer component folder
|
||||
- `--transformer-weights-path` for a quantized safetensors file, directory, or repo
|
||||
|
||||
See [Quantization](../quantization) for supported quantization families and examples.
|
||||
|
||||
## Configuration Files
|
||||
|
||||
Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file.
|
||||
|
||||
```bash Command
|
||||
sglang generate --config config.yaml
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```yaml Config
|
||||
model_path: FastVideo/FastHunyuan-diffusers
|
||||
prompt: A beautiful woman in a red dress walking down a street
|
||||
output_path: outputs/
|
||||
num_gpus: 2
|
||||
sp_size: 2
|
||||
tp_size: 1
|
||||
num_frames: 45
|
||||
height: 720
|
||||
width: 1280
|
||||
num_inference_steps: 6
|
||||
seed: 1024
|
||||
fps: 24
|
||||
precision: bf16
|
||||
vae_precision: fp16
|
||||
vae_tiling: true
|
||||
vae_sp: true
|
||||
enable_torch_compile: false
|
||||
```
|
||||
|
||||
## Generate
|
||||
|
||||
Run a one-off generation task without launching a persistent server. Pass both server arguments and sampling parameters after the `generate` subcommand:
|
||||
`sglang generate` runs a single generation job and exits when the job finishes.
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
)
|
||||
|
||||
SAMPLING_ARGS=(
|
||||
--prompt "A curious raccoon"
|
||||
--save-output
|
||||
--output-path outputs
|
||||
--output-file-name "A curious raccoon.mp4"
|
||||
)
|
||||
|
||||
sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
```
|
||||
|
||||
You can also enable Cache-DiT acceleration via an environment variable:
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
```bash Command
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--text-encoder-cpu-offload \
|
||||
--pin-cpu-memory \
|
||||
--num-gpus 4 \
|
||||
--ulysses-degree 2 \
|
||||
--ring-degree 2 \
|
||||
--prompt "A curious raccoon" \
|
||||
--save-output \
|
||||
--output-path outputs \
|
||||
--output-file-name "a-curious-raccoon.mp4"
|
||||
```
|
||||
|
||||
<Note>
|
||||
HTTP server-related arguments are ignored in `generate` mode. The process shuts down automatically once generation completes.
|
||||
HTTP server-only arguments are ignored by `sglang generate`.
|
||||
</Note>
|
||||
|
||||
For diffusers pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../cache_dit).
|
||||
|
||||
## Serve
|
||||
|
||||
Launch the SGLang Diffusion HTTP server and interact through the OpenAI-compatible API.
|
||||
`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
)
|
||||
|
||||
sglang serve "${SERVER_ARGS[@]}"
|
||||
```
|
||||
|
||||
- `--model-path` -- which model to load (e.g. `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`)
|
||||
- `--port` -- HTTP port to listen on (default: `30010`)
|
||||
|
||||
For full API usage including image/video generation and LoRA management, see the [OpenAI API documentation](./openai-api).
|
||||
|
||||
---
|
||||
|
||||
## Supported arguments
|
||||
|
||||
### Server arguments
|
||||
|
||||
<Accordion title="Server arguments reference">
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--model-path MODEL_PATH` | Path to the model or HuggingFace model ID |
|
||||
| `--lora-path LORA_PATH` | Path to a LoRA adapter (local or HuggingFace ID). If omitted, LoRA is not applied |
|
||||
| `--lora-nickname NAME` | Nickname for the LoRA adapter (default: `default`) |
|
||||
| `--num-gpus NUM` | Number of GPUs to use |
|
||||
| `--tp-size SIZE` | Tensor parallelism size (encoder only; keep at most 1 when text encoder offload is enabled) |
|
||||
| `--sp-degree SIZE` | Sequence parallelism size (typically should match the number of GPUs) |
|
||||
| `--ulysses-degree SIZE` | DeepSpeed-Ulysses-style SP degree in USP |
|
||||
| `--ring-degree SIZE` | Ring attention-style SP degree in USP |
|
||||
| `--attention-backend BACKEND` | Attention backend. Native pipelines: `fa`, `torch_sdpa`, `sage_attn`, etc. Diffusers pipelines: `flash`, `_flash_3_hub`, `sage`, `xformers` |
|
||||
| `--attention-backend-config CONFIG` | Config for the attention backend. Accepts a JSON string, a JSON/YAML file path, or `key=value` pairs |
|
||||
| `--cache-dit-config PATH` | Path to a Cache-DiT YAML/JSON config (diffusers backend only) |
|
||||
| `--dit-precision DTYPE` | Precision for the DiT model (`fp32`, `fp16`, `bf16`) |
|
||||
| `--text-encoder-cpu-offload` | Offload text encoders to CPU |
|
||||
| `--pin-cpu-memory` | Pin CPU memory for faster transfers |
|
||||
|
||||
</Accordion>
|
||||
|
||||
### Sampling parameters
|
||||
|
||||
<Accordion title="Generation parameters">
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--prompt PROMPT` | Text description for the image or video to generate |
|
||||
| `--negative-prompt PROMPT` | Negative prompt to guide generation away from certain concepts |
|
||||
| `--num-inference-steps STEPS` | Number of denoising steps |
|
||||
| `--seed SEED` | Random seed for reproducible generation |
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Image/video configuration">
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--height HEIGHT` | Height of the generated output |
|
||||
| `--width WIDTH` | Width of the generated output |
|
||||
| `--num-frames NUM` | Number of frames to generate (video only) |
|
||||
| `--fps FPS` | Frames per second for the saved output (video only) |
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Output options">
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--save-output` | Save the image or video to disk |
|
||||
| `--output-path PATH` | Directory to save the generated output |
|
||||
| `--output-file-name NAME` | File name for the saved output |
|
||||
| `--return-frames` | Return the raw frames instead of saving |
|
||||
|
||||
</Accordion>
|
||||
|
||||
### Frame interpolation (video only)
|
||||
|
||||
Frame interpolation is a post-processing step that synthesizes new frames between each pair of consecutive generated frames, producing smoother motion without re-running the diffusion model.
|
||||
|
||||
The `--frame-interpolation-exp` flag controls how many rounds of interpolation to apply: each round inserts one new frame into every gap between adjacent frames, so the output frame count follows the formula:
|
||||
|
||||
$$
|
||||
\text{output frames} = (N - 1) \times 2^{\text{exp}} + 1
|
||||
$$
|
||||
|
||||
For example, 5 original frames with `exp=1` -> 4 gaps x 1 new frame + 5 originals = **9 frames**; with `exp=2` -> **17 frames**.
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--enable-frame-interpolation` | Enable frame interpolation. Model weights are downloaded automatically on first use |
|
||||
| `--frame-interpolation-exp EXP` | Interpolation exponent -- `1` = 2x temporal resolution, `2` = 4x, etc. (default: `1`) |
|
||||
| `--frame-interpolation-scale SCALE` | RIFE inference scale; use `0.5` for high-resolution inputs to save memory (default: `1.0`) |
|
||||
| `--frame-interpolation-model-path PATH` | Local directory or HuggingFace repo ID containing RIFE `flownet.pkl` weights (default: `elfgum/RIFE-4.22.lite`, downloaded automatically) |
|
||||
|
||||
**Example** -- generate a 5-frame video and interpolate to 9 frames ($(5 - 1) \times 2^1 + 1 = 9$):
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--prompt "A dog running through a park" \
|
||||
--num-frames 5 \
|
||||
--enable-frame-interpolation \
|
||||
--frame-interpolation-exp 1 \
|
||||
--save-output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration files
|
||||
|
||||
Instead of passing every parameter on the command line, you can use a JSON or YAML config file. Command-line arguments take precedence over config values.
|
||||
|
||||
```bash
|
||||
sglang generate --config config.json
|
||||
```
|
||||
|
||||
<Tabs>
|
||||
<Tab title="JSON">
|
||||
```json config.json
|
||||
{
|
||||
"model_path": "FastVideo/FastHunyuan-diffusers",
|
||||
"prompt": "A beautiful woman in a red dress walking down a street",
|
||||
"output_path": "outputs/",
|
||||
"num_gpus": 2,
|
||||
"sp_size": 2,
|
||||
"tp_size": 1,
|
||||
"num_frames": 45,
|
||||
"height": 720,
|
||||
"width": 1280,
|
||||
"num_inference_steps": 6,
|
||||
"seed": 1024,
|
||||
"fps": 24,
|
||||
"precision": "bf16",
|
||||
"vae_precision": "fp16",
|
||||
"vae_tiling": true,
|
||||
"vae_sp": true,
|
||||
"vae_config": {
|
||||
"load_encoder": false,
|
||||
"load_decoder": true,
|
||||
"tile_sample_min_height": 256,
|
||||
"tile_sample_min_width": 256
|
||||
},
|
||||
"text_encoder_precisions": ["fp16", "fp16"],
|
||||
"mask_strategy_file_path": null,
|
||||
"enable_torch_compile": false
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="YAML">
|
||||
```yaml config.yaml
|
||||
model_path: "FastVideo/FastHunyuan-diffusers"
|
||||
prompt: "A beautiful woman in a red dress walking down a street"
|
||||
output_path: "outputs/"
|
||||
num_gpus: 2
|
||||
sp_size: 2
|
||||
tp_size: 1
|
||||
num_frames: 45
|
||||
height: 720
|
||||
width: 1280
|
||||
num_inference_steps: 6
|
||||
seed: 1024
|
||||
fps: 24
|
||||
precision: "bf16"
|
||||
vae_precision: "fp16"
|
||||
vae_tiling: true
|
||||
vae_sp: true
|
||||
vae_config:
|
||||
load_encoder: false
|
||||
load_decoder: true
|
||||
tile_sample_min_height: 256
|
||||
tile_sample_min_width: 256
|
||||
text_encoder_precisions:
|
||||
- "fp16"
|
||||
- "fp16"
|
||||
mask_strategy_file_path: null
|
||||
enable_torch_compile: false
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
To see all available options:
|
||||
|
||||
```bash
|
||||
sglang generate --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Component path overrides
|
||||
|
||||
You can override any pipeline component (e.g. `vae`, `transformer`, `text_encoder`) by specifying a custom checkpoint path with `--<component>-path`, where `<component>` matches the key in the model's `model_index.json`.
|
||||
|
||||
### Example: FLUX.2-dev with Tiny AutoEncoder
|
||||
|
||||
Replace the default VAE with a distilled tiny autoencoder for ~3x faster decoding:
|
||||
|
||||
```bash
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path=black-forest-labs/FLUX.2-dev \
|
||||
--vae-path=fal/FLUX.2-Tiny-AutoEncoder
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--text-encoder-cpu-offload \
|
||||
--pin-cpu-memory \
|
||||
--num-gpus 4 \
|
||||
--ulysses-degree 2 \
|
||||
--ring-degree 2 \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
You can also use a local path:
|
||||
### Cloud Storage
|
||||
|
||||
```bash
|
||||
SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation.
|
||||
|
||||
```bash Command
|
||||
export SGLANG_CLOUD_STORAGE_TYPE=s3
|
||||
export SGLANG_S3_BUCKET_NAME=my-bucket
|
||||
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
|
||||
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
|
||||
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
|
||||
```
|
||||
|
||||
See [Environment Variables](../environment_variables) for the full set of storage options.
|
||||
|
||||
## Component Path Overrides
|
||||
|
||||
Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `--<component>-path`.
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path=black-forest-labs/FLUX.2-dev \
|
||||
--vae-path=~/.cache/huggingface/hub/models--fal--FLUX.2-Tiny-AutoEncoder/snapshots/.../vae
|
||||
--model-path black-forest-labs/FLUX.2-dev \
|
||||
--vae-path fal/FLUX.2-Tiny-AutoEncoder
|
||||
```
|
||||
|
||||
<Warning>
|
||||
The component key must match the one in the model's `model_index.json` (e.g. `vae`).
|
||||
The path must be either a HuggingFace repo ID or point to a complete component folder containing `config.json` and safetensors files.
|
||||
</Warning>
|
||||
The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory.
|
||||
|
||||
---
|
||||
## Diffusers Backend
|
||||
|
||||
## Diffusers backend
|
||||
Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class.
|
||||
|
||||
SGLang Diffusion supports a diffusers backend that runs any diffusers-compatible model through SGLang's infrastructure using vanilla diffusers pipelines. This is useful for models without native SGLang implementations or models with custom pipeline classes.
|
||||
### Key Options
|
||||
|
||||
### Backend arguments
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Argument</th>
|
||||
<th>Values</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>--backend</code></td>
|
||||
<td><code>auto</code>, <code>sglang</code>, <code>diffusers</code></td>
|
||||
<td>Choose native SGLang, force native, or force diffusers</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--diffusers-attention-backend</code></td>
|
||||
<td><code>flash</code>, <code>_flash_3_hub</code>, <code>sage</code>, <code>xformers</code>, <code>native</code></td>
|
||||
<td>Attention backend for diffusers pipelines</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--trust-remote-code</code></td>
|
||||
<td>flag</td>
|
||||
<td>Required for models with custom pipeline classes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--vae-tiling</code> and <code>--vae-slicing</code></td>
|
||||
<td>flag</td>
|
||||
<td>Lower memory usage for VAE decode</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--dit-precision</code> and <code>--vae-precision</code></td>
|
||||
<td><code>fp16</code>, <code>bf16</code>, <code>fp32</code></td>
|
||||
<td>Precision controls</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--enable-torch-compile</code></td>
|
||||
<td>flag</td>
|
||||
<td>Enable <code>torch.compile</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--cache-dit-config</code></td>
|
||||
<td><code>{PATH}</code></td>
|
||||
<td>Cache-DiT config for diffusers pipelines</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
| Argument | Values | Description |
|
||||
|:--|:--|:--|
|
||||
| `--backend` | `auto` (default), `sglang`, `diffusers` | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline |
|
||||
| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines |
|
||||
| `--trust-remote-code` | flag | Required for models with custom pipeline classes |
|
||||
| `--vae-tiling` | flag | Enable VAE tiling for large image support (decodes tile-by-tile) |
|
||||
| `--vae-slicing` | flag | Enable VAE slicing for lower memory usage (decodes slice-by-slice) |
|
||||
| `--dit-precision` | `fp16`, `bf16`, `fp32` | Precision for the diffusion transformer |
|
||||
| `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision for the VAE |
|
||||
|
||||
### Example: running Ovis-Image-7B
|
||||
|
||||
[Ovis-Image-7B](https://huggingface.co/AIDC-AI/Ovis-Image-7B) is a 7B text-to-image model optimized for high-quality text rendering.
|
||||
### Example
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
@@ -308,59 +268,4 @@ sglang generate \
|
||||
--output-file-name ovis_garden.png
|
||||
```
|
||||
|
||||
### Extra diffusers arguments
|
||||
|
||||
For pipeline-specific parameters not exposed via CLI, use `diffusers_kwargs` in a config file:
|
||||
|
||||
```json config.json
|
||||
{
|
||||
"model_path": "AIDC-AI/Ovis-Image-7B",
|
||||
"backend": "diffusers",
|
||||
"prompt": "A beautiful landscape",
|
||||
"diffusers_kwargs": {
|
||||
"cross_attention_kwargs": {"scale": 0.5}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
sglang generate --config config.json
|
||||
```
|
||||
|
||||
### Cache-DiT acceleration
|
||||
|
||||
Users on the diffusers backend can leverage Cache-DiT acceleration by loading custom cache configs from a YAML file. See the [Cache-DiT documentation](../cache-dit) for details.
|
||||
|
||||
---
|
||||
|
||||
## Cloud storage support
|
||||
|
||||
The server supports automatically uploading generated artifacts to S3-compatible cloud storage (AWS S3, MinIO, Alibaba Cloud OSS, Tencent Cloud COS).
|
||||
|
||||
The workflow is: **Generate -> Upload -> Delete local file**. The API response returns the public URL of the uploaded object.
|
||||
|
||||
1. **Install boto3**
|
||||
|
||||
```bash
|
||||
pip install boto3
|
||||
```
|
||||
|
||||
2. **Set environment variables**
|
||||
|
||||
```bash
|
||||
export SGLANG_CLOUD_STORAGE_TYPE=s3
|
||||
export SGLANG_S3_BUCKET_NAME=my-bucket
|
||||
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
|
||||
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
|
||||
|
||||
# Optional: custom endpoint for MinIO/OSS/COS
|
||||
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
|
||||
```
|
||||
|
||||
3. **Launch the server**
|
||||
|
||||
```bash
|
||||
sglang serve --model-path MODEL_PATH
|
||||
```
|
||||
|
||||
See the [environment variables reference](../environment-variables) for all storage-related variables.
|
||||
For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file.
|
||||
|
||||
@@ -1,421 +0,0 @@
|
||||
---
|
||||
title: OpenAI API
|
||||
sidebarTitle: OpenAI API
|
||||
description: Image and video generation endpoints with LoRA adapter management.
|
||||
---
|
||||
|
||||
The SGLang Diffusion HTTP server implements an OpenAI-compatible API for image and video generation, as well as dynamic LoRA adapter management.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.11+ if you plan to use the OpenAI Python SDK.
|
||||
- A running SGLang Diffusion server (see the [CLI reference](./cli) for launch instructions).
|
||||
|
||||
## Start the server
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
--port 30010
|
||||
)
|
||||
|
||||
sglang serve "${SERVER_ARGS[@]}"
|
||||
```
|
||||
|
||||
- `--model-path` -- path to the model or HuggingFace model ID
|
||||
- `--port` -- HTTP port to listen on (default: `30000`)
|
||||
|
||||
### Get model information
|
||||
|
||||
**Endpoint:** `GET /models`
|
||||
|
||||
Returns model path, task type, pipeline configuration, and precision settings.
|
||||
|
||||
<CodeGroup>
|
||||
```bash curl
|
||||
curl -sS -X GET "http://localhost:30010/models"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
|
||||
"task_type": "T2V",
|
||||
"pipeline_name": "wan_pipeline",
|
||||
"pipeline_class": "WanPipeline",
|
||||
"num_gpus": 4,
|
||||
"dit_precision": "bf16",
|
||||
"vae_precision": "fp16"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Image generation
|
||||
|
||||
The server implements an OpenAI-compatible Images API under the `/v1/images` namespace.
|
||||
|
||||
### Create an image
|
||||
|
||||
**Endpoint:** `POST /v1/images/generations`
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
|
||||
|
||||
img = client.images.generate(
|
||||
prompt="A calico cat playing a piano on stage",
|
||||
size="1024x1024",
|
||||
n=1,
|
||||
response_format="b64_json",
|
||||
)
|
||||
|
||||
image_bytes = base64.b64decode(img.data[0].b64_json)
|
||||
with open("output.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
```bash curl
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/generations" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A calico cat playing a piano on stage",
|
||||
"size": "1024x1024",
|
||||
"n": 1,
|
||||
"response_format": "b64_json"
|
||||
}'
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
If `response_format=url` is used and cloud storage is not configured, the API returns a relative URL like `/v1/images/<IMAGE_ID>/content`.
|
||||
</Note>
|
||||
|
||||
### Edit an image
|
||||
|
||||
**Endpoint:** `POST /v1/images/edits`
|
||||
|
||||
Accepts a multipart form upload with input images and a text prompt. Returns either a base64-encoded image or a URL.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="b64_json response">
|
||||
```bash
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "image=@local_input_image.png" \
|
||||
-F "url=image_url.jpg" \
|
||||
-F "prompt=A calico cat playing a piano on stage" \
|
||||
-F "size=1024x1024" \
|
||||
-F "response_format=b64_json"
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="URL response">
|
||||
```bash
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "image=@local_input_image.png" \
|
||||
-F "url=image_url.jpg" \
|
||||
-F "prompt=A calico cat playing a piano on stage" \
|
||||
-F "size=1024x1024" \
|
||||
-F "response_format=url"
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Download image content
|
||||
|
||||
When `response_format=url` is used, the API returns a relative URL like `/v1/images/<IMAGE_ID>/content`.
|
||||
|
||||
**Endpoint:** `GET /v1/images/{image_id}/content`
|
||||
|
||||
```bash
|
||||
curl -sS -L "http://localhost:30010/v1/images/<IMAGE_ID>/content" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-o output.png
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Video generation
|
||||
|
||||
The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace.
|
||||
|
||||
### Create a video
|
||||
|
||||
**Endpoint:** `POST /v1/videos`
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
|
||||
|
||||
video = client.videos.create(
|
||||
prompt="A calico cat playing a piano on stage",
|
||||
size="1280x720"
|
||||
)
|
||||
print(f"Video ID: {video.id}, Status: {video.status}")
|
||||
```
|
||||
|
||||
```bash curl
|
||||
curl -sS -X POST "http://localhost:30010/v1/videos" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A calico cat playing a piano on stage",
|
||||
"size": "1280x720"
|
||||
}'
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### List videos
|
||||
|
||||
**Endpoint:** `GET /v1/videos`
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
videos = client.videos.list()
|
||||
for item in videos.data:
|
||||
print(item.id, item.status)
|
||||
```
|
||||
|
||||
```bash curl
|
||||
curl -sS -X GET "http://localhost:30010/v1/videos" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Download video content
|
||||
|
||||
**Endpoint:** `GET /v1/videos/{video_id}/content`
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
import time
|
||||
|
||||
# Poll for completion
|
||||
while True:
|
||||
page = client.videos.list()
|
||||
item = next((v for v in page.data if v.id == video_id), None)
|
||||
if item and item.status == "completed":
|
||||
break
|
||||
time.sleep(5)
|
||||
|
||||
# Download content
|
||||
resp = client.videos.download_content(video_id=video_id)
|
||||
with open("output.mp4", "wb") as f:
|
||||
f.write(resp.read())
|
||||
```
|
||||
|
||||
```bash curl
|
||||
curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-o output.mp4
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
---
|
||||
|
||||
## LoRA management
|
||||
|
||||
The server supports dynamic loading, merging, and unmerging of LoRA adapters.
|
||||
|
||||
<Info>
|
||||
- **Mutual exclusion:** Only one LoRA can be merged (active) at a time.
|
||||
- **Switching:** To switch LoRAs, you must first unmerge the current one, then set the new one.
|
||||
- **Caching:** The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has negligible cost.
|
||||
</Info>
|
||||
|
||||
### Set LoRA adapter
|
||||
|
||||
Loads one or more LoRA adapters and merges their weights into the model. Supports both single LoRA (backward compatible) and multiple LoRA adapters.
|
||||
|
||||
**Endpoint:** `POST /v1/set_lora`
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|:--|:--|:--|
|
||||
| `lora_nickname` | string or list | A unique identifier for the LoRA adapter(s). Required |
|
||||
| `lora_path` | string or list | Path to `.safetensors` file(s) or HuggingFace repo ID(s). Required for first load; optional when re-activating a cached nickname |
|
||||
| `target` | string or list | Which transformer(s) to apply the LoRA to: `"all"` (default), `"transformer"`, `"transformer_2"`, `"critic"` |
|
||||
| `strength` | float or list | LoRA strength for merge (default: `1.0`). Values < 1.0 reduce the effect, > 1.0 amplify it |
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Single LoRA">
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": "lora_name",
|
||||
"lora_path": "/path/to/lora.safetensors",
|
||||
"target": "all",
|
||||
"strength": 0.8
|
||||
}'
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Multiple LoRAs">
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": ["lora_1", "lora_2"],
|
||||
"lora_path": ["/path/to/lora1.safetensors", "/path/to/lora2.safetensors"],
|
||||
"target": ["transformer", "transformer_2"],
|
||||
"strength": [0.8, 1.0]
|
||||
}'
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Same target">
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": ["style_lora", "character_lora"],
|
||||
"lora_path": ["/path/to/style.safetensors", "/path/to/character.safetensors"],
|
||||
"target": "all",
|
||||
"strength": [0.7, 0.9]
|
||||
}'
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
<Note>
|
||||
When using multiple LoRAs:
|
||||
- All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length.
|
||||
- If `target` or `strength` is a single value, it will be applied to all LoRAs.
|
||||
- Multiple LoRAs applied to the same target will be merged in order.
|
||||
</Note>
|
||||
|
||||
### Merge LoRA weights
|
||||
|
||||
Manually merges the currently set LoRA weights into the base model.
|
||||
|
||||
**Endpoint:** `POST /v1/merge_lora_weights`
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|:--|:--|:--|
|
||||
| `target` | string | Which transformer(s) to merge: `"all"` (default), `"transformer"`, `"transformer_2"`, `"critic"` |
|
||||
| `strength` | float | LoRA strength for merge (default: `1.0`) |
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/merge_lora_weights \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"strength": 0.8}'
|
||||
```
|
||||
|
||||
<Tip>
|
||||
`set_lora` automatically performs a merge, so this endpoint is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.
|
||||
</Tip>
|
||||
|
||||
### Unmerge LoRA weights
|
||||
|
||||
Unmerges the currently active LoRA weights from the base model, restoring it to its original state. Call this before setting a different LoRA.
|
||||
|
||||
**Endpoint:** `POST /v1/unmerge_lora_weights`
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/unmerge_lora_weights \
|
||||
-H "Content-Type: application/json"
|
||||
```
|
||||
|
||||
### List LoRA adapters
|
||||
|
||||
Returns loaded LoRA adapters and current application status per module.
|
||||
|
||||
**Endpoint:** `GET /v1/list_loras`
|
||||
|
||||
```bash
|
||||
curl -sS -X GET "http://localhost:30010/v1/list_loras"
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"loaded_adapters": [
|
||||
{ "nickname": "lora_a", "path": "/weights/lora_a.safetensors" },
|
||||
{ "nickname": "lora_b", "path": "/weights/lora_b.safetensors" }
|
||||
],
|
||||
"active": {
|
||||
"transformer": [
|
||||
{
|
||||
"nickname": "lora2",
|
||||
"path": "tarn59/pixel_art_style_lora_z_image_turbo",
|
||||
"merged": true,
|
||||
"strength": 1.0
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example: switching LoRAs
|
||||
|
||||
1. **Set LoRA A**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-d '{"lora_nickname": "lora_a", "lora_path": "path/to/A"}'
|
||||
```
|
||||
|
||||
2. **Generate with LoRA A**
|
||||
|
||||
Run your image or video generation requests.
|
||||
|
||||
3. **Unmerge LoRA A**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/unmerge_lora_weights
|
||||
```
|
||||
|
||||
4. **Set LoRA B**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-d '{"lora_nickname": "lora_b", "lora_path": "path/to/B"}'
|
||||
```
|
||||
|
||||
5. **Generate with LoRA B**
|
||||
|
||||
Run your image or video generation requests with the new adapter.
|
||||
|
||||
---
|
||||
|
||||
## Output quality
|
||||
|
||||
Control output quality and compression for both image and video generation through the `output-quality` and `output-compression` parameters.
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|:--|:--|:--|
|
||||
| `output-quality` | string | Preset quality level. Default: `"default"` |
|
||||
| `output-compression` | integer | Direct compression level override (0-100). When provided, takes precedence over `output-quality` |
|
||||
|
||||
**Quality presets:**
|
||||
|
||||
| Preset | Compression value |
|
||||
|:--|:--|
|
||||
| `"maximum"` | 100 |
|
||||
| `"high"` | 90 |
|
||||
| `"medium"` | 55 |
|
||||
| `"low"` | 35 |
|
||||
| `"default"` | Auto (50 for video, 75 for image) |
|
||||
|
||||
<Warning>
|
||||
- When both `output-quality` and `output-compression` are provided, `output-compression` takes precedence.
|
||||
- Quality settings apply to JPEG and video formats. PNG uses lossless compression and ignores these settings.
|
||||
- Lower compression values (or `"low"` quality preset) produce smaller files but may show visible artifacts.
|
||||
</Warning>
|
||||
@@ -0,0 +1,450 @@
|
||||
---
|
||||
title: OpenAI API
|
||||
sidebarTitle: OpenAI API
|
||||
description: Image and video generation endpoints with LoRA adapter management.
|
||||
---
|
||||
The SGLang diffusion HTTP server implements an OpenAI-compatible API for image and video generation, as well as LoRA adapter management.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.11+ if you plan to use the OpenAI Python SDK.
|
||||
|
||||
## Serve
|
||||
|
||||
Launch the server using the `sglang serve` command.
|
||||
|
||||
### Start the server
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
--port 30010
|
||||
)
|
||||
|
||||
sglang serve "${SERVER_ARGS[@]}"
|
||||
```
|
||||
|
||||
- **--model-path**: Path to the model or model ID.
|
||||
- **--port**: HTTP port to listen on (default: `30000`).
|
||||
|
||||
**Get Model Information**
|
||||
|
||||
**Endpoint:** `GET /models`
|
||||
|
||||
Returns information about the model served by this server, including model path, task type, pipeline configuration, and precision settings.
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash curl
|
||||
curl -sS -X GET "http://localhost:30010/models"
|
||||
```
|
||||
|
||||
**Response Example:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
|
||||
"task_type": "T2V",
|
||||
"pipeline_name": "wan_pipeline",
|
||||
"pipeline_class": "WanPipeline",
|
||||
"num_gpus": 4,
|
||||
"dit_precision": "bf16",
|
||||
"vae_precision": "fp16"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Image Generation
|
||||
|
||||
The server implements an OpenAI-compatible Images API under the `/v1/images` namespace.
|
||||
|
||||
**Create an image**
|
||||
|
||||
**Endpoint:** `POST /v1/images/generations`
|
||||
|
||||
**Python Example (b64_json response):**
|
||||
|
||||
```python Python
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
|
||||
|
||||
img = client.images.generate(
|
||||
prompt="A calico cat playing a piano on stage",
|
||||
size="1024x1024",
|
||||
n=1,
|
||||
response_format="b64_json",
|
||||
)
|
||||
|
||||
image_bytes = base64.b64decode(img.data[0].b64_json)
|
||||
with open("output.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash curl
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/generations" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A calico cat playing a piano on stage",
|
||||
"size": "1024x1024",
|
||||
"n": 1,
|
||||
"response_format": "b64_json"
|
||||
}'
|
||||
```
|
||||
|
||||
> **Note**
|
||||
> If `response_format=url` is used and cloud storage is not configured, the API returns
|
||||
> a relative URL like `/v1/images/<IMAGE_ID>/content`.
|
||||
|
||||
**Edit an image**
|
||||
|
||||
**Endpoint:** `POST /v1/images/edits`
|
||||
|
||||
This endpoint accepts a multipart form upload with input images and a text prompt. The server can return either a base64-encoded image or a URL to download the image.
|
||||
|
||||
**Curl Example (b64_json response):**
|
||||
|
||||
```bash Command
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "image=@local_input_image.png" \
|
||||
-F "url=image_url.jpg" \
|
||||
-F "prompt=A calico cat playing a piano on stage" \
|
||||
-F "size=1024x1024" \
|
||||
-F "response_format=b64_json"
|
||||
```
|
||||
|
||||
**Curl Example (URL response):**
|
||||
|
||||
```bash Command
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "image=@local_input_image.png" \
|
||||
-F "url=image_url.jpg" \
|
||||
-F "prompt=A calico cat playing a piano on stage" \
|
||||
-F "size=1024x1024" \
|
||||
-F "response_format=url"
|
||||
```
|
||||
|
||||
**Download image content**
|
||||
|
||||
When `response_format=url` is used with `POST /v1/images/generations` or `POST /v1/images/edits`,
|
||||
the API returns a relative URL like `/v1/images/<IMAGE_ID>/content`.
|
||||
|
||||
**Endpoint:** `GET /v1/images/{image_id}/content`
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -L "http://localhost:30010/v1/images/<IMAGE_ID>/content" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-o output.png
|
||||
```
|
||||
|
||||
### Video Generation
|
||||
|
||||
The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace.
|
||||
|
||||
**Create a video (text-to-video)**
|
||||
|
||||
**Endpoint:** `POST /v1/videos`
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
|
||||
|
||||
video = client.videos.create(
|
||||
prompt="A calico cat playing a piano on stage",
|
||||
size="1280x720"
|
||||
)
|
||||
print(f"Video ID: {video.id}, Status: {video.status}")
|
||||
```
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash curl
|
||||
curl -sS -X POST "http://localhost:30010/v1/videos" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A calico cat playing a piano on stage",
|
||||
"size": "1280x720"
|
||||
}'
|
||||
```
|
||||
|
||||
**Create a video (image-to-video)**
|
||||
|
||||
For I2V or TI2V models (e.g., Wan2.1 I2V, LTX-2.3 two-stage), pass an input image via multipart form upload or a reference URL.
|
||||
|
||||
**Curl Example (multipart form upload):**
|
||||
|
||||
```bash Command
|
||||
curl -sS -X POST "http://localhost:30010/v1/videos" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "prompt=A cat playing a piano" \
|
||||
-F "input_reference=@input_image.png" \
|
||||
-F "size=1280x720"
|
||||
```
|
||||
|
||||
**Curl Example (reference URL):**
|
||||
|
||||
```bash Command
|
||||
curl -sS -X POST "http://localhost:30010/v1/videos" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A cat playing a piano",
|
||||
"reference_url": "https://example.com/input_image.png",
|
||||
"size": "1280x720"
|
||||
}'
|
||||
```
|
||||
|
||||
**List videos**
|
||||
|
||||
**Endpoint:** `GET /v1/videos`
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Python
|
||||
videos = client.videos.list()
|
||||
for item in videos.data:
|
||||
print(item.id, item.status)
|
||||
```
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash curl
|
||||
curl -sS -X GET "http://localhost:30010/v1/videos" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890"
|
||||
```
|
||||
|
||||
**Download video content**
|
||||
|
||||
**Endpoint:** `GET /v1/videos/{video_id}/content`
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Python
|
||||
import time
|
||||
|
||||
# Poll for completion
|
||||
while True:
|
||||
page = client.videos.list()
|
||||
item = next((v for v in page.data if v.id == video_id), None)
|
||||
if item and item.status == "completed":
|
||||
break
|
||||
time.sleep(5)
|
||||
|
||||
# Download content
|
||||
resp = client.videos.download_content(video_id=video_id)
|
||||
with open("output.mp4", "wb") as f:
|
||||
f.write(resp.read())
|
||||
```
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash curl
|
||||
curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-o output.mp4
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### LoRA Management
|
||||
|
||||
The server supports dynamic loading, merging, and unmerging of LoRA adapters.
|
||||
|
||||
**Important Notes:**
|
||||
- Mutual Exclusion: Only one LoRA can be *merged* (active) at a time
|
||||
- Switching: To switch LoRAs, you must first `unmerge` the current one, then `set` the new one
|
||||
- Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost
|
||||
|
||||
**Set LoRA Adapter**
|
||||
|
||||
Loads one or more LoRA adapters and merges their weights into the model. Supports both single LoRA (backward compatible) and multiple LoRA adapters.
|
||||
|
||||
**Endpoint:** `POST /v1/set_lora`
|
||||
|
||||
**Parameters:**
|
||||
- `lora_nickname` (string or list of strings, required): A unique identifier for the LoRA adapter(s). Can be a single string or a list of strings for multiple LoRAs
|
||||
- `lora_path` (string or list of strings/None, optional): Path to the `.safetensors` file(s) or Hugging Face repo ID(s). Required for the first load; optional if re-activating a cached nickname. If a list, must match the length of `lora_nickname`
|
||||
- `target` (string or list of strings, optional): Which transformer(s) to apply the LoRA to. If a list, must match the length of `lora_nickname`. Valid values:
|
||||
- `"all"` (default): Apply to all transformers
|
||||
- `"transformer"`: Apply only to the primary transformer (high noise for Wan2.2)
|
||||
- `"transformer_2"`: Apply only to transformer_2 (low noise for Wan2.2)
|
||||
- `"critic"`: Apply only to the critic model
|
||||
- `strength` (float or list of floats, optional): LoRA strength for merge, default 1.0. If a list, must match the length of `lora_nickname`. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
|
||||
|
||||
**Single LoRA Example:**
|
||||
|
||||
```bash Command
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": "lora_name",
|
||||
"lora_path": "/path/to/lora.safetensors",
|
||||
"target": "all",
|
||||
"strength": 0.8
|
||||
}'
|
||||
```
|
||||
|
||||
**Multiple LoRA Example:**
|
||||
|
||||
```bash Command
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": ["lora_1", "lora_2"],
|
||||
"lora_path": ["/path/to/lora1.safetensors", "/path/to/lora2.safetensors"],
|
||||
"target": ["transformer", "transformer_2"],
|
||||
"strength": [0.8, 1.0]
|
||||
}'
|
||||
```
|
||||
|
||||
**Multiple LoRA with Same Target:**
|
||||
|
||||
```bash Command
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": ["style_lora", "character_lora"],
|
||||
"lora_path": ["/path/to/style.safetensors", "/path/to/character.safetensors"],
|
||||
"target": "all",
|
||||
"strength": [0.7, 0.9]
|
||||
}'
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> When using multiple LoRAs:
|
||||
> - All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length
|
||||
> - If `target` or `strength` is a single value, it will be applied to all LoRAs
|
||||
> - Multiple LoRAs applied to the same target will be merged in order
|
||||
|
||||
|
||||
**Merge LoRA Weights**
|
||||
|
||||
Manually merges the currently set LoRA weights into the base model.
|
||||
|
||||
> [!NOTE]
|
||||
> `set_lora` automatically performs a merge, so this is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.*
|
||||
|
||||
**Endpoint:** `POST /v1/merge_lora_weights`
|
||||
|
||||
**Parameters:**
|
||||
- `target` (string, optional): Which transformer(s) to merge. One of "all" (default), "transformer", "transformer_2", "critic"
|
||||
- `strength` (float, optional): LoRA strength for merge, default 1.0. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/merge_lora_weights \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"strength": 0.8}'
|
||||
```
|
||||
|
||||
|
||||
**Unmerge LoRA Weights**
|
||||
|
||||
Unmerges the currently active LoRA weights from the base model, restoring it to its original state. This **must** be called before setting a different LoRA.
|
||||
|
||||
**Endpoint:** `POST /v1/unmerge_lora_weights`
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/unmerge_lora_weights \
|
||||
-H "Content-Type: application/json"
|
||||
```
|
||||
|
||||
**List LoRA Adapters**
|
||||
|
||||
Returns loaded LoRA adapters and current application status per module.
|
||||
|
||||
**Endpoint:** `GET /v1/list_loras`
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash
|
||||
curl -sS -X GET "http://localhost:30010/v1/list_loras"
|
||||
```
|
||||
|
||||
**Response Example:**
|
||||
|
||||
```json
|
||||
{
|
||||
"loaded_adapters": [
|
||||
{ "nickname": "lora_a", "path": "/weights/lora_a.safetensors" },
|
||||
{ "nickname": "lora_b", "path": "/weights/lora_b.safetensors" }
|
||||
],
|
||||
"active": {
|
||||
"transformer": [
|
||||
{
|
||||
"nickname": "lora2",
|
||||
"path": "tarn59/pixel_art_style_lora_z_image_turbo",
|
||||
"merged": true,
|
||||
"strength": 1.0
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
- If LoRA is not enabled for the current pipeline, the server will return an error.
|
||||
- `num_lora_layers_with_weights` counts only layers that have LoRA weights applied for the active adapter.
|
||||
|
||||
### Example: Switching LoRAs
|
||||
|
||||
1. Set LoRA A:
|
||||
```bash Command
|
||||
curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_a", "lora_path": "path/to/A"}'
|
||||
```
|
||||
2. Generate with LoRA A...
|
||||
3. Unmerge LoRA A:
|
||||
```bash Command
|
||||
curl -X POST http://localhost:30010/v1/unmerge_lora_weights
|
||||
```
|
||||
4. Set LoRA B:
|
||||
```bash Command
|
||||
curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_b", "lora_path": "path/to/B"}'
|
||||
```
|
||||
5. Generate with LoRA B...
|
||||
|
||||
### Adjust Output Quality
|
||||
|
||||
The server supports adjusting output quality and compression levels for both image and video generation through the `output-quality` and `output-compression` parameters.
|
||||
|
||||
#### Parameters
|
||||
|
||||
- **`output-quality`** (string, optional): Preset quality level that automatically sets compression. **Default is `"default"`**. Valid values:
|
||||
- `"maximum"`: Highest quality (100)
|
||||
- `"high"`: High quality (90)
|
||||
- `"medium"`: Medium quality (55)
|
||||
- `"low"`: Lower quality (35)
|
||||
- `"default"`: Auto-adjust based on media type (50 for video, 75 for image)
|
||||
|
||||
- **`output-compression`** (integer, optional): Direct compression level override (0-100). **Default is `None`**. When provided (not `None`), takes precedence over `output-quality`.
|
||||
- `0`: Lowest quality, smallest file size
|
||||
- `100`: Highest quality, largest file size
|
||||
|
||||
#### Notes
|
||||
|
||||
- **Precedence**: When both `output-quality` and `output-compression` are provided, `output-compression` takes precedence
|
||||
- **Format Support**: Quality settings apply to JPEG, and video formats. PNG uses lossless compression and ignores these settings
|
||||
- **File Size vs Quality**: Lower compression values (or "low" quality preset) produce smaller files but may show visible artifacts
|
||||
@@ -0,0 +1,237 @@
|
||||
---
|
||||
title: "Post-Processing"
|
||||
metatags:
|
||||
description: "Use SGLang Diffusion post-processing for frame interpolation and spatial upscaling after generation."
|
||||
---
|
||||
|
||||
SGLang diffusion supports optional post-processing steps that run after
|
||||
generation to improve temporal smoothness (frame interpolation) or spatial
|
||||
resolution (upscaling). These steps are independent of the diffusion model and
|
||||
can be combined in a single run.
|
||||
|
||||
When both are enabled, **frame interpolation runs first** (increasing the frame
|
||||
count), then **upscaling runs on every frame** (increasing the spatial
|
||||
resolution).
|
||||
|
||||
---
|
||||
|
||||
## Frame Interpolation (video only)
|
||||
|
||||
Frame interpolation synthesizes new frames between each pair of consecutive
|
||||
generated frames, producing smoother motion without re-running the diffusion
|
||||
model.
|
||||
|
||||
The `--frame-interpolation-exp` flag controls how many rounds of interpolation
|
||||
to apply: each round inserts one new frame into every gap between adjacent
|
||||
frames, so the output frame count follows the formula:
|
||||
|
||||
> **(N − 1) × 2^exp + 1**
|
||||
>
|
||||
> e.g. 5 original frames with `exp=1` → 4 gaps × 1 new frame + 5 originals = **9** frames;
|
||||
> with `exp=2` → **17** frames.
|
||||
|
||||
### CLI Arguments
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Argument</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>--enable-frame-interpolation</code></td>
|
||||
<td>Enable frame interpolation. Model weights are downloaded automatically on first use.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--frame-interpolation-exp {EXP}</code></td>
|
||||
<td>Interpolation exponent — <code>1</code> = 2× temporal resolution, <code>2</code> = 4×, etc. (default: <code>1</code>)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--frame-interpolation-scale {SCALE}</code></td>
|
||||
<td>RIFE inference scale; use <code>0.5</code> for high-resolution inputs to save memory (default: <code>1.0</code>)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--frame-interpolation-model-path {PATH}</code></td>
|
||||
<td>Local directory or HuggingFace repo ID containing RIFE <code>flownet.pkl</code> weights (default: <code>elfgum/RIFE-4.22.lite</code>, downloaded automatically)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Supported Models
|
||||
|
||||
Frame interpolation uses the [RIFE](https://github.com/hzwer/Practical-RIFE)
|
||||
(Real-Time Intermediate Flow Estimation) architecture. Only **RIFE 4.22.lite**
|
||||
(`IFNet` with 4-scale `IFBlock` backbone) is supported. The network topology is
|
||||
hard-coded, so custom weights provided via `--frame-interpolation-model-path`
|
||||
must be a `flownet.pkl` checkpoint that is compatible with this architecture.
|
||||
|
||||
Other RIFE versions (e.g., older `v4.x` variants with different block counts)
|
||||
or entirely different frame interpolation methods (FILM, AMT, etc.) are **not
|
||||
supported**.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.33%"}} />
|
||||
<col style={{width: "33.33%"}} />
|
||||
<col style={{width: "33.33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Weight</th>
|
||||
<th>HuggingFace Repo</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>RIFE 4.22.lite *(default)*</td>
|
||||
<td><a href="https://huggingface.co/elfgum/RIFE-4.22.lite"><code>elfgum/RIFE-4.22.lite</code></a></td>
|
||||
<td>Lightweight model, downloaded automatically on first use</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Example
|
||||
|
||||
Generate a 5-frame video and interpolate to 9 frames ((5 − 1) × 2¹ + 1 = 9):
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--prompt "A dog running through a park" \
|
||||
--num-frames 5 \
|
||||
--enable-frame-interpolation \
|
||||
--frame-interpolation-exp 1 \
|
||||
--save-output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Upscaling (image and video)
|
||||
|
||||
Upscaling increases the spatial resolution of generated images or video frames
|
||||
using [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN). The model weights
|
||||
are downloaded automatically on first use and cached for subsequent runs.
|
||||
|
||||
### CLI Arguments
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Argument</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>--enable-upscaling</code></td>
|
||||
<td>Enable post-generation upscaling using Real-ESRGAN.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--upscaling-scale {SCALE}</code></td>
|
||||
<td>Desired upscaling factor (default: <code>4</code>). The 4× model is used internally; if a different scale is requested, a bicubic resize is applied after the network output.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--upscaling-model-path {PATH}</code></td>
|
||||
<td>Local <code>.pth</code> file, HuggingFace repo ID, or <code>repo_id:filename</code> for Real-ESRGAN weights (default: <code>ai-forever/Real-ESRGAN</code> with <code>RealESRGAN_x4.pth</code>, downloaded automatically). Use the <code>repo_id:filename</code> format to specify a custom weight file from a HuggingFace repo (e.g. <code>my-org/my-esrgan:weights.pth</code>).</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Supported Models
|
||||
|
||||
Upscaling supports two Real-ESRGAN network architectures. The correct
|
||||
architecture is **auto-detected** from the checkpoint keys, so you only need to
|
||||
point `--upscaling-model-path` at a valid `.pth` file:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.33%"}} />
|
||||
<col style={{width: "33.33%"}} />
|
||||
<col style={{width: "33.33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Architecture</th>
|
||||
<th>Example Weights</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>RRDBNet</strong></td>
|
||||
<td><code>RealESRGAN_x4plus.pth</code></td>
|
||||
<td>Heavier model with higher quality; best for photos</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>SRVGGNetCompact</strong></td>
|
||||
<td><code>RealESRGAN_x4.pth</code> *(default)*, <code>realesr-animevideov3.pth</code>, <code>realesr-general-x4v3.pth</code></td>
|
||||
<td>Lightweight model; faster inference, good for video</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The default weight file is
|
||||
[`ai-forever/Real-ESRGAN`](https://huggingface.co/ai-forever/Real-ESRGAN) with
|
||||
`RealESRGAN_x4.pth` (SRVGGNetCompact, 4× native scale).
|
||||
|
||||
Other super-resolution models (e.g., SwinIR, HAT, BSRGAN) are **not supported**
|
||||
— only Real-ESRGAN checkpoints using the two architectures above are
|
||||
compatible.
|
||||
|
||||
### Examples
|
||||
|
||||
Generate a 1024×1024 image and upscale to 4096×4096:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path black-forest-labs/FLUX.2-dev \
|
||||
--prompt "A cat sitting on a windowsill" \
|
||||
--output-size 1024x1024 \
|
||||
--enable-upscaling \
|
||||
--save-output
|
||||
```
|
||||
|
||||
Generate a video and upscale each frame by 4×:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--prompt "A curious raccoon" \
|
||||
--enable-upscaling \
|
||||
--upscaling-scale 4 \
|
||||
--save-output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Combining Frame Interpolation and Upscaling
|
||||
|
||||
Frame interpolation and upscaling can be combined in a single run.
|
||||
Interpolation is applied first (increasing the frame count), then upscaling is
|
||||
applied to every frame (increasing the spatial resolution).
|
||||
|
||||
Example — generate 5 frames, interpolate to 9 frames, and upscale each frame
|
||||
by 4×:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--prompt "A curious raccoon" \
|
||||
--num-frames 5 \
|
||||
--enable-frame-interpolation \
|
||||
--frame-interpolation-exp 1 \
|
||||
--enable-upscaling \
|
||||
--upscaling-scale 4 \
|
||||
--save-output
|
||||
```
|
||||
Reference in New Issue
Block a user