minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] Sync docs_new with legacy docs and update migration redirects (#23337)

Browse Source 
Co-authored-by: Mingyi <wisclmy0611@gmail.com>
This commit is contained in:
zijiexia
2026-04-21 00:15:17 -07:00
committed by GitHub
co-authored by Mingyi
parent f63def8510
commit 900aad5f72
179 changed files with 16014 additions and 8162 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/sglang-diffusion/api/cli.mdx
+222 -317
View File
@@ -3,295 +3,255 @@ title: CLI reference
sidebarTitle: CLI
description: Run one-off generation tasks and launch the HTTP server from the command line.
---
Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`.
The `sglang` CLI provides two main subcommands for diffusion inference:
### Overlay repos for non-diffusers models
- **`sglang generate`** -- run a one-off generation without a persistent server
- **`sglang serve`** -- launch the OpenAI-compatible HTTP server
If `--model-path` points to a supported non-diffusers source repo, SGLang can resolve it
through a self-hosted overlay repo.
## Prerequisites
SGLang first checks a built-in overlay registry. Concrete built-in mappings can be added over time without changing the CLI surface.
A working SGLang Diffusion installation with the `sglang` CLI available in your `$PATH`. See the [installation guide](../installation) for setup instructions.
Override example:
```bash Command
export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{
"Wan-AI/Wan2.2-S2V-14B": {
"overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay",
"overlay_revision": "main"
}
}'
sglang generate \
--model-path Wan-AI/Wan2.2-S2V-14B \
--config configs/wan_s2v.yaml
```
The overlay repo should be a complete diffusers-style/componentized repo
You can also pass the overlay repo itself as `--model-path` if it contains `_overlay/overlay_manifest.json`.
Notes:
1. `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is only an optional override for
development and debugging. It accepts either a JSON object or a path to a JSON
file, and can extend or replace built-in entries for the current process.
2. On the first load, SGLang will:
- download overlay metadata from the overlay repo
- download the required files from the original source repo
- materialize a local standard component repo under `~/.cache/sgl_diffusion/materialized_models/`
3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory.
## Quick Start
### Generate
```bash Command
sglang generate \
--model-path Qwen/Qwen-Image \
--prompt "A beautiful sunset over the mountains" \
--save-output
```
### Serve
```bash Command
sglang serve \
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
--num-gpus 4 \
--ulysses-degree 2 \
--ring-degree 2 \
--port 30010
```
For request and response examples, see [OpenAI-Compatible API](./openai_api).
<Tip>
Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags.
</Tip>
## Common Options
### Model and runtime
- `--model-path &#123;MODEL&#125;`: model path or Hugging Face model ID
- `--lora-path &#123;PATH&#125;` and `--lora-nickname &#123;NAME&#125;`: load a LoRA adapter
- `--num-gpus &#123;N&#125;`: number of GPUs to use
- `--tp-size &#123;N&#125;`: tensor parallelism size, mainly for encoders
- `--sp-degree &#123;N&#125;`: sequence parallelism size
- `--ulysses-degree &#123;N&#125;` and `--ring-degree &#123;N&#125;`: USP parallelism controls
- `--attention-backend &#123;BACKEND&#125;`: attention backend for native SGLang pipelines
- `--attention-backend-config &#123;CONFIG&#125;`: attention backend configuration
### Sampling and output
- `--prompt &#123;PROMPT&#125;` and `--negative-prompt &#123;PROMPT&#125;`
- `--image-path &#123;PATH&#125; [&#123;PATH&#125; ...]`: input image(s) for image-to-video or image-to-image generation
- `--num-inference-steps &#123;STEPS&#125;` and `--seed &#123;SEED&#125;`
- `--height &#123;HEIGHT&#125;`, `--width &#123;WIDTH&#125;`, `--num-frames &#123;N&#125;`, `--fps &#123;FPS&#125;`
- `--output-path &#123;PATH&#125;`, `--output-file-name &#123;NAME&#125;`, `--save-output`, `--return-frames`
For frame interpolation and upscaling, see [Post-Processing](./post_processing).
### Quantized transformers
For quantized transformer checkpoints, prefer:
- `--model-path` for the base pipeline
- `--transformer-path` for a quantized `transformers` transformer component folder
- `--transformer-weights-path` for a quantized safetensors file, directory, or repo
See [Quantization](../quantization) for supported quantization families and examples.
## Configuration Files
Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file.
```bash Command
sglang generate --config config.yaml
```
Example:
```yaml Config
model_path: FastVideo/FastHunyuan-diffusers
prompt: A beautiful woman in a red dress walking down a street
output_path: outputs/
num_gpus: 2
sp_size: 2
tp_size: 1
num_frames: 45
height: 720
width: 1280
num_inference_steps: 6
seed: 1024
fps: 24
precision: bf16
vae_precision: fp16
vae_tiling: true
vae_sp: true
enable_torch_compile: false
```
## Generate
Run a one-off generation task without launching a persistent server. Pass both server arguments and sampling parameters after the `generate` subcommand:
`sglang generate` runs a single generation job and exits when the job finishes.
```bash
SERVER_ARGS=(
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
--text-encoder-cpu-offload
--pin-cpu-memory
--num-gpus 4
--ulysses-degree=2
--ring-degree=2
)
SAMPLING_ARGS=(
--prompt "A curious raccoon"
--save-output
--output-path outputs
--output-file-name "A curious raccoon.mp4"
)
sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
```
You can also enable Cache-DiT acceleration via an environment variable:
```bash
SGLANG_CACHE_DIT_ENABLED=true sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
```bash Command
sglang generate \
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--text-encoder-cpu-offload \
--pin-cpu-memory \
--num-gpus 4 \
--ulysses-degree 2 \
--ring-degree 2 \
--prompt "A curious raccoon" \
--save-output \
--output-path outputs \
--output-file-name "a-curious-raccoon.mp4"
```
<Note>
HTTP server-related arguments are ignored in `generate` mode. The process shuts down automatically once generation completes.
HTTP server-only arguments are ignored by `sglang generate`.
</Note>
For diffusers pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../cache_dit).
## Serve
Launch the SGLang Diffusion HTTP server and interact through the OpenAI-compatible API.
`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.
```bash
SERVER_ARGS=(
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
--text-encoder-cpu-offload
--pin-cpu-memory
--num-gpus 4
--ulysses-degree=2
--ring-degree=2
)
sglang serve "${SERVER_ARGS[@]}"
```
- `--model-path` -- which model to load (e.g. `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`)
- `--port` -- HTTP port to listen on (default: `30010`)
For full API usage including image/video generation and LoRA management, see the [OpenAI API documentation](./openai-api).
---
## Supported arguments
### Server arguments
<Accordion title="Server arguments reference">
| Argument | Description |
|:--|:--|
| `--model-path MODEL_PATH` | Path to the model or HuggingFace model ID |
| `--lora-path LORA_PATH` | Path to a LoRA adapter (local or HuggingFace ID). If omitted, LoRA is not applied |
| `--lora-nickname NAME` | Nickname for the LoRA adapter (default: `default`) |
| `--num-gpus NUM` | Number of GPUs to use |
| `--tp-size SIZE` | Tensor parallelism size (encoder only; keep at most 1 when text encoder offload is enabled) |
| `--sp-degree SIZE` | Sequence parallelism size (typically should match the number of GPUs) |
| `--ulysses-degree SIZE` | DeepSpeed-Ulysses-style SP degree in USP |
| `--ring-degree SIZE` | Ring attention-style SP degree in USP |
| `--attention-backend BACKEND` | Attention backend. Native pipelines: `fa`, `torch_sdpa`, `sage_attn`, etc. Diffusers pipelines: `flash`, `_flash_3_hub`, `sage`, `xformers` |
| `--attention-backend-config CONFIG` | Config for the attention backend. Accepts a JSON string, a JSON/YAML file path, or `key=value` pairs |
| `--cache-dit-config PATH` | Path to a Cache-DiT YAML/JSON config (diffusers backend only) |
| `--dit-precision DTYPE` | Precision for the DiT model (`fp32`, `fp16`, `bf16`) |
| `--text-encoder-cpu-offload` | Offload text encoders to CPU |
| `--pin-cpu-memory` | Pin CPU memory for faster transfers |
</Accordion>
### Sampling parameters
<Accordion title="Generation parameters">
| Argument | Description |
|:--|:--|
| `--prompt PROMPT` | Text description for the image or video to generate |
| `--negative-prompt PROMPT` | Negative prompt to guide generation away from certain concepts |
| `--num-inference-steps STEPS` | Number of denoising steps |
| `--seed SEED` | Random seed for reproducible generation |
</Accordion>
<Accordion title="Image/video configuration">
| Argument | Description |
|:--|:--|
| `--height HEIGHT` | Height of the generated output |
| `--width WIDTH` | Width of the generated output |
| `--num-frames NUM` | Number of frames to generate (video only) |
| `--fps FPS` | Frames per second for the saved output (video only) |
</Accordion>
<Accordion title="Output options">
| Argument | Description |
|:--|:--|
| `--save-output` | Save the image or video to disk |
| `--output-path PATH` | Directory to save the generated output |
| `--output-file-name NAME` | File name for the saved output |
| `--return-frames` | Return the raw frames instead of saving |
</Accordion>
### Frame interpolation (video only)
Frame interpolation is a post-processing step that synthesizes new frames between each pair of consecutive generated frames, producing smoother motion without re-running the diffusion model.
The `--frame-interpolation-exp` flag controls how many rounds of interpolation to apply: each round inserts one new frame into every gap between adjacent frames, so the output frame count follows the formula:
$$
\text{output frames} = (N - 1) \times 2^{\text{exp}} + 1
$$
For example, 5 original frames with `exp=1` -> 4 gaps x 1 new frame + 5 originals = **9 frames**; with `exp=2` -> **17 frames**.
| Argument | Description |
|:--|:--|
| `--enable-frame-interpolation` | Enable frame interpolation. Model weights are downloaded automatically on first use |
| `--frame-interpolation-exp EXP` | Interpolation exponent -- `1` = 2x temporal resolution, `2` = 4x, etc. (default: `1`) |
| `--frame-interpolation-scale SCALE` | RIFE inference scale; use `0.5` for high-resolution inputs to save memory (default: `1.0`) |
| `--frame-interpolation-model-path PATH` | Local directory or HuggingFace repo ID containing RIFE `flownet.pkl` weights (default: `elfgum/RIFE-4.22.lite`, downloaded automatically) |
**Example** -- generate a 5-frame video and interpolate to 9 frames ($(5 - 1) \times 2^1 + 1 = 9$):
```bash
sglang generate \
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--prompt "A dog running through a park" \
--num-frames 5 \
--enable-frame-interpolation \
--frame-interpolation-exp 1 \
--save-output
```
---
## Configuration files
Instead of passing every parameter on the command line, you can use a JSON or YAML config file. Command-line arguments take precedence over config values.
```bash
sglang generate --config config.json
```
<Tabs>
<Tab title="JSON">
```json config.json
{
"model_path": "FastVideo/FastHunyuan-diffusers",
"prompt": "A beautiful woman in a red dress walking down a street",
"output_path": "outputs/",
"num_gpus": 2,
"sp_size": 2,
"tp_size": 1,
"num_frames": 45,
"height": 720,
"width": 1280,
"num_inference_steps": 6,
"seed": 1024,
"fps": 24,
"precision": "bf16",
"vae_precision": "fp16",
"vae_tiling": true,
"vae_sp": true,
"vae_config": {
"load_encoder": false,
"load_decoder": true,
"tile_sample_min_height": 256,
"tile_sample_min_width": 256
},
"text_encoder_precisions": ["fp16", "fp16"],
"mask_strategy_file_path": null,
"enable_torch_compile": false
}
```
</Tab>
<Tab title="YAML">
```yaml config.yaml
model_path: "FastVideo/FastHunyuan-diffusers"
prompt: "A beautiful woman in a red dress walking down a street"
output_path: "outputs/"
num_gpus: 2
sp_size: 2
tp_size: 1
num_frames: 45
height: 720
width: 1280
num_inference_steps: 6
seed: 1024
fps: 24
precision: "bf16"
vae_precision: "fp16"
vae_tiling: true
vae_sp: true
vae_config:
load_encoder: false
load_decoder: true
tile_sample_min_height: 256
tile_sample_min_width: 256
text_encoder_precisions:
- "fp16"
- "fp16"
mask_strategy_file_path: null
enable_torch_compile: false
```
</Tab>
</Tabs>
To see all available options:
```bash
sglang generate --help
```
---
## Component path overrides
You can override any pipeline component (e.g. `vae`, `transformer`, `text_encoder`) by specifying a custom checkpoint path with `--<component>-path`, where `<component>` matches the key in the model's `model_index.json`.
### Example: FLUX.2-dev with Tiny AutoEncoder
Replace the default VAE with a distilled tiny autoencoder for ~3x faster decoding:
```bash
```bash Command
sglang serve \
--model-path=black-forest-labs/FLUX.2-dev \
--vae-path=fal/FLUX.2-Tiny-AutoEncoder
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
--text-encoder-cpu-offload \
--pin-cpu-memory \
--num-gpus 4 \
--ulysses-degree 2 \
--ring-degree 2 \
--port 30010
```
You can also use a local path:
### Cloud Storage
```bash
SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation.
```bash Command
export SGLANG_CLOUD_STORAGE_TYPE=s3
export SGLANG_S3_BUCKET_NAME=my-bucket
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
```
See [Environment Variables](../environment_variables) for the full set of storage options.
## Component Path Overrides
Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `--<component>-path`.
```bash Command
sglang serve \
--model-path=black-forest-labs/FLUX.2-dev \
--vae-path=~/.cache/huggingface/hub/models--fal--FLUX.2-Tiny-AutoEncoder/snapshots/.../vae
--model-path black-forest-labs/FLUX.2-dev \
--vae-path fal/FLUX.2-Tiny-AutoEncoder
```
<Warning>
The component key must match the one in the model's `model_index.json` (e.g. `vae`).
The path must be either a HuggingFace repo ID or point to a complete component folder containing `config.json` and safetensors files.
</Warning>
The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory.
---
## Diffusers Backend
## Diffusers backend
Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class.
SGLang Diffusion supports a diffusers backend that runs any diffusers-compatible model through SGLang's infrastructure using vanilla diffusers pipelines. This is useful for models without native SGLang implementations or models with custom pipeline classes.
### Key Options
### Backend arguments
<table>
<thead>
<tr>
<th>Argument</th>
<th>Values</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>--backend</code></td>
<td><code>auto</code>, <code>sglang</code>, <code>diffusers</code></td>
<td>Choose native SGLang, force native, or force diffusers</td>
</tr>
<tr>
<td><code>--diffusers-attention-backend</code></td>
<td><code>flash</code>, <code>_flash_3_hub</code>, <code>sage</code>, <code>xformers</code>, <code>native</code></td>
<td>Attention backend for diffusers pipelines</td>
</tr>
<tr>
<td><code>--trust-remote-code</code></td>
<td>flag</td>
<td>Required for models with custom pipeline classes</td>
</tr>
<tr>
<td><code>--vae-tiling</code> and <code>--vae-slicing</code></td>
<td>flag</td>
<td>Lower memory usage for VAE decode</td>
</tr>
<tr>
<td><code>--dit-precision</code> and <code>--vae-precision</code></td>
<td><code>fp16</code>, <code>bf16</code>, <code>fp32</code></td>
<td>Precision controls</td>
</tr>
<tr>
<td><code>--enable-torch-compile</code></td>
<td>flag</td>
<td>Enable <code>torch.compile</code></td>
</tr>
<tr>
<td><code>--cache-dit-config</code></td>
<td><code>&#123;PATH&#125;</code></td>
<td>Cache-DiT config for diffusers pipelines</td>
</tr>
</tbody>
</table>
| Argument | Values | Description |
|:--|:--|:--|
| `--backend` | `auto` (default), `sglang`, `diffusers` | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline |
| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines |
| `--trust-remote-code` | flag | Required for models with custom pipeline classes |
| `--vae-tiling` | flag | Enable VAE tiling for large image support (decodes tile-by-tile) |
| `--vae-slicing` | flag | Enable VAE slicing for lower memory usage (decodes slice-by-slice) |
| `--dit-precision` | `fp16`, `bf16`, `fp32` | Precision for the diffusion transformer |
| `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision for the VAE |
### Example: running Ovis-Image-7B
[Ovis-Image-7B](https://huggingface.co/AIDC-AI/Ovis-Image-7B) is a 7B text-to-image model optimized for high-quality text rendering.
### Example
```bash
sglang generate \
@@ -308,59 +268,4 @@ sglang generate \
--output-file-name ovis_garden.png
```
### Extra diffusers arguments
For pipeline-specific parameters not exposed via CLI, use `diffusers_kwargs` in a config file:
```json config.json
{
"model_path": "AIDC-AI/Ovis-Image-7B",
"backend": "diffusers",
"prompt": "A beautiful landscape",
"diffusers_kwargs": {
"cross_attention_kwargs": {"scale": 0.5}
}
}
```
```bash
sglang generate --config config.json
```
### Cache-DiT acceleration
Users on the diffusers backend can leverage Cache-DiT acceleration by loading custom cache configs from a YAML file. See the [Cache-DiT documentation](../cache-dit) for details.
---
## Cloud storage support
The server supports automatically uploading generated artifacts to S3-compatible cloud storage (AWS S3, MinIO, Alibaba Cloud OSS, Tencent Cloud COS).
The workflow is: **Generate -> Upload -> Delete local file**. The API response returns the public URL of the uploaded object.
1. **Install boto3**
```bash
pip install boto3
```
2. **Set environment variables**
```bash
export SGLANG_CLOUD_STORAGE_TYPE=s3
export SGLANG_S3_BUCKET_NAME=my-bucket
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
# Optional: custom endpoint for MinIO/OSS/COS
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
```
3. **Launch the server**
```bash
sglang serve --model-path MODEL_PATH
```
See the [environment variables reference](../environment-variables) for all storage-related variables.
For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file.