[Docs] Add Falcon H1, Hunyuan-Large, Qwen3-Omni support and update Diffusion usage (#17888)
Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: Ratish P <114130421+Ratish1@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
This commit is contained in:
Rishit Shivam
co-authored by
Rishitshivam
Ratish P
gemini-code-assist[bot]
Adarsh Shirawalmath
zhaochenyang20
Rishitshivam
Ratish P
gemini-code-assist[bot]
Adarsh Shirawalmath
zhaochenyang20
parent
f2e0048d06
commit
c850a8a41a
@@ -0,0 +1,19 @@
|
||||
# Diffusion
|
||||
|
||||
SGLang supports two categories of diffusion models for different use cases. This page covers image and video generation; for diffusion LLMs, see [Diffusion LLMs](diffusion_llms.md).
|
||||
|
||||
## Image & Video Generation Models
|
||||
|
||||
For generating images and videos from text prompts, SGLang supports [many](../supported_models/image_generation/diffusion_models.md#image-generation-models) models like:
|
||||
|
||||
- **FLUX, Qwen-Image** - High-quality image generation
|
||||
- **Wan 2.2, HunyuanVideo** - Video generation
|
||||
|
||||
```bash
|
||||
# Example: Launch FLUX for image generation
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path black-forest-labs/FLUX.2-klein-4B \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
**Full model list:** [Diffusion Models](../supported_models/image_generation/diffusion_models.md)
|
||||
@@ -0,0 +1,14 @@
|
||||
# Diffusion Language Models (dLLMs)
|
||||
|
||||
These are text-generation models that use diffusion (denoising) instead of autoregressive decoding:
|
||||
|
||||
- **LLaDA** - Large Language Diffusion with mAsking
|
||||
|
||||
```bash
|
||||
# Example: Launch LLaDA for text generation
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path GSAI-ML/LLaDA-8B-Instruct \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
**Full model list:** [Diffusion Language Models](../supported_models/text_generation/diffusion_language_models.md)
|
||||
@@ -9,7 +9,7 @@
|
||||
"SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
|
||||
"A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).\n",
|
||||
"\n",
|
||||
"This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported_models/embedding_models.md)\n"
|
||||
"This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported_models/retrieval_ranking/embedding_models.md)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -173,7 +173,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Multi-Modal Embedding Model\n",
|
||||
"Please refer to [Multi-Modal Embedding Model](../supported_models/embedding_models.md)"
|
||||
"Please refer to [Multi-Modal Embedding Model](../supported_models/retrieval_ranking/embedding_models.md)"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
"A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).\n",
|
||||
"This tutorial covers the vision APIs for vision language models.\n",
|
||||
"\n",
|
||||
"SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported_models/multimodal_language_models.md).\n",
|
||||
"SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported_models/text_generation/multimodal_language_models.md).\n",
|
||||
"\n",
|
||||
"As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py)."
|
||||
]
|
||||
|
||||
+8
-13
@@ -35,6 +35,8 @@ Its core features include:
|
||||
basic_usage/native_api.ipynb
|
||||
basic_usage/sampling_params.md
|
||||
basic_usage/popular_model_usage.rst
|
||||
basic_usage/diffusion_llms.md
|
||||
basic_usage/diffusion.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
@@ -67,21 +69,14 @@ Its core features include:
|
||||
advanced_features/sglang_for_rl.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:maxdepth: 2
|
||||
:caption: Supported Models
|
||||
|
||||
supported_models/generative_models.md
|
||||
supported_models/multimodal_language_models.md
|
||||
supported_models/diffusion_language_models.md
|
||||
supported_models/diffusion_models.md
|
||||
supported_models/embedding_models.md
|
||||
supported_models/reward_models.md
|
||||
supported_models/rerank_models.md
|
||||
supported_models/classify_models.md
|
||||
supported_models/support_new_models.md
|
||||
supported_models/transformers_fallback.md
|
||||
supported_models/modelscope.md
|
||||
supported_models/mindspore_models.md
|
||||
supported_models/text_generation/index
|
||||
supported_models/image_generation/index
|
||||
supported_models/retrieval_ranking/index
|
||||
supported_models/specialized/index
|
||||
supported_models/extending/index
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
@@ -8,5 +8,6 @@ Ascend NPUs
|
||||
ascend_npu_support_models.md
|
||||
ascend_npu_deepseek_example.md
|
||||
ascend_npu_qwen3_examples.md
|
||||
mindspore_backend.md
|
||||
ascend_contribution_guide.md
|
||||
ascend_npu_best_practice.md
|
||||
|
||||
@@ -6,8 +6,8 @@ MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc
|
||||
|
||||
## Requirements
|
||||
|
||||
MindSpore currently only supports Ascend NPU devices. Users need to first install CANN 8.5.
|
||||
The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com).
|
||||
MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages.
|
||||
The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2.
|
||||
|
||||
## Supported Models
|
||||
|
||||
@@ -19,7 +19,7 @@ Currently, the following models are supported:
|
||||
|
||||
## Installation
|
||||
|
||||
> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](../platforms/ascend_npu.md) and then install `sgl-mindspore`:
|
||||
> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](ascend_npu.md) and then install `sgl-mindspore`:
|
||||
|
||||
```shell
|
||||
git clone https://github.com/mindspore-lab/sgl-mindspore.git
|
||||
@@ -32,9 +32,9 @@ pip install -e .
|
||||
|
||||
Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example.
|
||||
|
||||
### Offline inference
|
||||
### Offline infer
|
||||
|
||||
Use the following script for offline inference:
|
||||
Use the following script for offline infer:
|
||||
|
||||
```python
|
||||
import sglang as sgl
|
||||
@@ -385,7 +385,7 @@
|
||||
"## Multi-modal Generation\n",
|
||||
"\n",
|
||||
"You may use SGLang frontend language to define multi-modal prompts.\n",
|
||||
"See [here](https://docs.sglang.io/supported_models/generative_models.html) for supported models."
|
||||
"See [here](https://docs.sglang.io/supported_models/text_generation/generative_models.html) for supported models."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
Extending SGLang
|
||||
================
|
||||
|
||||
Adding new models and alternative backends.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
support_new_models.md
|
||||
transformers_fallback.md
|
||||
modelscope.md
|
||||
+3
-3
@@ -70,7 +70,6 @@ file, test the new model on your local machine and report the results on demonst
|
||||
MMMU-Pro, etc.) in your PR. \\
|
||||
For VLMs, also include a test in `test_vision_openai_server_{x}.py` (e.g. [test_vision_openai_server_a.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server_a.py), [test_vision_openai_server_b.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server_b.py)).
|
||||
|
||||
|
||||
This is an example command to run to test a new model on your local machine:
|
||||
|
||||
```bash
|
||||
@@ -223,6 +222,7 @@ After receiving the logits for the next token, we can finally perform our biasin
|
||||
|
||||
return res
|
||||
```
|
||||
|
||||
Now, our `LlamaWrapper` model is created and ready to be served!
|
||||
|
||||
### Serving Our Model Via SGLang's Offline Engine
|
||||
@@ -310,9 +310,9 @@ if __name__ == "__main__":
|
||||
|
||||
Now, when we call `python run.py`, we will get the outputs of our newly created model!
|
||||
|
||||
|
||||
## Documentation
|
||||
Add to table of supported models in [generative_models.md](https://github.com/sgl-project/sglang/blob/main/docs/supported_models/generative_models.md) or [multimodal_language_models.md](https://github.com/sgl-project/sglang/blob/main/docs/supported_models/multimodal_language_models.md)
|
||||
|
||||
Add to table of supported models in [generative_models.md](../text_generation/generative_models.md) or [multimodal_language_models.md](../text_generation/multimodal_language_models.md)
|
||||
|
||||
---
|
||||
|
||||
+34
-29
@@ -1,5 +1,7 @@
|
||||
# Diffusion Models
|
||||
|
||||
> This page covers **image and video generation**. For **text generation** using diffusion LLMs (e.g., LLaDA2.0), see [Diffusion Language Models](../text_generation/diffusion_language_models.md).
|
||||
|
||||
SGLang Diffusion is an inference framework for accelerated image and video generation using diffusion models. It provides an end-to-end unified pipeline with optimized kernels from sgl-kernel and an efficient scheduler loop.
|
||||
|
||||
## Key Features
|
||||
@@ -93,7 +95,7 @@ default parameters when initializing and generating videos.
|
||||
### Video Generation Models
|
||||
|
||||
| Model Name | Hugging Face Model ID | Resolutions | TeaCache | Sliding Tile Attn | Sage Attn | Video Sparse Attention (VSA) |
|
||||
|:-----------------------------|:--------------------------------------------------|:--------------------|:--------:|:-----------------:|:---------:|:----------------------------:|
|
||||
| :--------------------------- | :------------------------------------------------ | :------------------ | :------: | :---------------: | :-------: | :--------------------------: |
|
||||
| FastWan2.1 T2V 1.3B | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` | 480p | ⭕ | ⭕ | ⭕ | ✅ |
|
||||
| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720p | ⭕ | ⭕ | ⭕ | ✅ |
|
||||
| Wan2.2 TI2V 5B | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | 720p | ⭕ | ⭕ | ✅ | ⭕ |
|
||||
@@ -111,7 +113,7 @@ default parameters when initializing and generating videos.
|
||||
### Image Generation Models
|
||||
|
||||
| Model Name | HuggingFace Model ID | Resolutions |
|
||||
|:-----------------|:----------------------------------------|:---------------|
|
||||
| :-------------- | :---------------------------------- | :------------- |
|
||||
| FLUX.1-dev | `black-forest-labs/FLUX.1-dev` | Any resolution |
|
||||
| FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | Any resolution |
|
||||
| FLUX.2-Klein | `black-forest-labs/FLUX.2-klein-4B` | Any resolution |
|
||||
@@ -133,7 +135,7 @@ This section lists example LoRAs that have been explicitly tested and verified w
|
||||
### Verified LoRAs by Base Model
|
||||
|
||||
| Base Model | Supported LoRAs |
|
||||
|:-----------------|:----------------|
|
||||
| :-------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Wan2.2 | `lightx2v/Wan2.2-Distill-Loras`<br>`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1` |
|
||||
| Wan2.1 | `lightx2v/Wan2.1-Distill-Loras` |
|
||||
| Z-Image-Turbo | `tarn59/pixel_art_style_lora_z_image_turbo`<br>`wcde/Z-Image-Turbo-DeJPEG-Lora` |
|
||||
@@ -146,7 +148,6 @@ This section lists example LoRAs that have been explicitly tested and verified w
|
||||
> [!NOTE]
|
||||
> Sliding Tile Attention: Currently, only Hopper GPUs (H100s) are supported.
|
||||
|
||||
|
||||
---
|
||||
|
||||
# SGLang diffusion CLI Inference
|
||||
@@ -158,7 +159,6 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
|
||||
- A working SGLang diffusion installation and the `sglang` CLI available in `$PATH`.
|
||||
- Python 3.11+ if you plan to use the OpenAI Python SDK.
|
||||
|
||||
|
||||
## Supported Arguments
|
||||
|
||||
### Server Arguments
|
||||
@@ -173,7 +173,6 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
|
||||
- `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP
|
||||
|
||||
|
||||
### Sampling Parameters
|
||||
|
||||
- `--prompt {PROMPT}`: Text description for the video you want to generate
|
||||
@@ -181,7 +180,6 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
|
||||
- `--negative-prompt {PROMPT}`: Negative prompt to guide generation away from certain concepts
|
||||
- `--seed {SEED}`: Random seed for reproducible generation
|
||||
|
||||
|
||||
#### Image/Video Configuration
|
||||
|
||||
- `--height {HEIGHT}`: Height of the generated output
|
||||
@@ -189,7 +187,6 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
|
||||
- `--num-frames {NUM_FRAMES}`: Number of frames to generate
|
||||
- `--fps {FPS}`: Frames per second for the saved output, if this is a video-generation task
|
||||
|
||||
|
||||
#### Output Options
|
||||
|
||||
- `--output-path {PATH}`: Directory to save the generated video
|
||||
@@ -232,10 +229,7 @@ Example configuration file (config.json):
|
||||
"tile_sample_min_height": 256,
|
||||
"tile_sample_min_width": 256
|
||||
},
|
||||
"text_encoder_precisions": [
|
||||
"fp16",
|
||||
"fp16"
|
||||
],
|
||||
"text_encoder_precisions": ["fp16", "fp16"],
|
||||
"mask_strategy_file_path": null,
|
||||
"enable_torch_compile": false
|
||||
}
|
||||
@@ -272,7 +266,6 @@ mask_strategy_file_path: null
|
||||
enable_torch_compile: false
|
||||
```
|
||||
|
||||
|
||||
To see all the options, you can use the `--help` flag:
|
||||
|
||||
```bash
|
||||
@@ -305,7 +298,6 @@ sglang serve "${SERVER_ARGS[@]}"
|
||||
|
||||
For detailed API usage, including Image, Video Generation and LoRA management, please refer to the [OpenAI API Documentation](#sglang-diffusion-openai-api).
|
||||
|
||||
|
||||
## Generate
|
||||
|
||||
Run a one-off generation task without launching a persistent server.
|
||||
@@ -347,7 +339,7 @@ SGLang diffusion supports a **diffusers backend** that allows you to run any dif
|
||||
### Arguments
|
||||
|
||||
| Argument | Values | Description |
|
||||
|----------|--------|-------------|
|
||||
| ------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `--backend` | `auto` (default), `sglang`, `diffusers` | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline. |
|
||||
| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines. See [diffusers attention backends](https://huggingface.co/docs/diffusers/main/en/optimization/attention_backends). |
|
||||
| `--trust-remote-code` | flag | Required for models with custom pipeline classes (e.g., Ovis). |
|
||||
@@ -385,7 +377,7 @@ For pipeline-specific parameters not exposed via CLI, use `diffusers_kwargs` in
|
||||
"backend": "diffusers",
|
||||
"prompt": "A beautiful landscape",
|
||||
"diffusers_kwargs": {
|
||||
"cross_attention_kwargs": {"scale": 0.5}
|
||||
"cross_attention_kwargs": { "scale": 0.5 }
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -633,7 +625,8 @@ curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
|
||||
The server supports dynamic loading, merging, and unmerging of LoRA adapters.
|
||||
|
||||
**Important Notes:**
|
||||
- Mutual Exclusion: Only one LoRA can be *merged* (active) at a time
|
||||
|
||||
- Mutual Exclusion: Only one LoRA can be _merged_ (active) at a time
|
||||
- Switching: To switch LoRAs, you must first `unmerge` the current one, then `set` the new one
|
||||
- Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost
|
||||
|
||||
@@ -644,6 +637,7 @@ Loads one or more LoRA adapters and merges their weights into the model. Support
|
||||
**Endpoint:** `POST /v1/set_lora`
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `lora_nickname` (string or list of strings, required): A unique identifier for the LoRA adapter(s). Can be a single string or a list of strings for multiple LoRAs
|
||||
- `lora_path` (string or list of strings/None, optional): Path to the `.safetensors` file(s) or Hugging Face repo ID(s). Required for the first load; optional if re-activating a cached nickname. If a list, must match the length of `lora_nickname`
|
||||
- `target` (string or list of strings, optional): Which transformer(s) to apply the LoRA to. If a list, must match the length of `lora_nickname`. Valid values:
|
||||
@@ -694,21 +688,22 @@ curl -X POST http://localhost:30010/v1/set_lora \
|
||||
|
||||
> [!NOTE]
|
||||
> When using multiple LoRAs:
|
||||
>
|
||||
> - All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length
|
||||
> - If `target` or `strength` is a single value, it will be applied to all LoRAs
|
||||
> - Multiple LoRAs applied to the same target will be merged in order
|
||||
|
||||
|
||||
#### Merge LoRA Weights
|
||||
|
||||
Manually merges the currently set LoRA weights into the base model.
|
||||
|
||||
> [!NOTE]
|
||||
> `set_lora` automatically performs a merge, so this is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.*
|
||||
> `set_lora` automatically performs a merge, so this is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.\*
|
||||
|
||||
**Endpoint:** `POST /v1/merge_lora_weights`
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `target` (string, optional): Which transformer(s) to merge. One of "all" (default), "transformer", "transformer_2", "critic"
|
||||
- `strength` (float, optional): LoRA strength for merge, default 1.0. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
|
||||
|
||||
@@ -720,7 +715,6 @@ curl -X POST http://localhost:30010/v1/merge_lora_weights \
|
||||
-d '{"strength": 0.8}'
|
||||
```
|
||||
|
||||
|
||||
#### Unmerge LoRA Weights
|
||||
|
||||
Unmerges the currently active LoRA weights from the base model, restoring it to its original state. This **must** be called before setting a different LoRA.
|
||||
@@ -768,6 +762,7 @@ curl -sS -X GET "http://localhost:30010/v1/list_loras"
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- If LoRA is not enabled for the current pipeline, the server will return an error.
|
||||
- `num_lora_layers_with_weights` counts only layers that have LoRA weights applied for the active adapter.
|
||||
|
||||
@@ -809,7 +804,7 @@ Backend selection is performed by the shared attention layers (e.g. `LocalAttent
|
||||
The CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
|
||||
|
||||
| CLI value | Enum value | Notes |
|
||||
|---|---|---|
|
||||
| -------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `fa` / `fa3` / `fa4` | `FA` | FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.__post_init__`). |
|
||||
| `torch_sdpa` | `TORCH_SDPA` | PyTorch `scaled_dot_product_attention`. |
|
||||
| `sliding_tile_attn` | `SLIDING_TILE_ATTN` | Sliding Tile Attention (STA). Requires `st_attn` and a mask-strategy config file set via the `SGLANG_DIFFUSION_ATTENTION_CONFIG` environment variable. |
|
||||
@@ -830,7 +825,7 @@ The selection order in `runtime/layers/attention/selector.py` is:
|
||||
## Platform support matrix
|
||||
|
||||
| Backend | CUDA | ROCm | MPS | Notes |
|
||||
|---|---:|---:|---:|---|
|
||||
| ------------------- | ---: | ---: | --: | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `fa` | ✅ | ✅ | ❌ | CUDA requires SM80+ and fp16/bf16. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. |
|
||||
| `torch_sdpa` | ✅ | ✅ | ✅ | Most compatible option across platforms. |
|
||||
| `sliding_tile_attn` | ✅ | ❌ | ❌ | CUDA-only. Requires `st_attn` and `SGLANG_DIFFUSION_ATTENTION_CONFIG`. |
|
||||
@@ -906,7 +901,7 @@ sglang generate --model-path Qwen/Qwen-Image \
|
||||
DBCache controls block-level caching behavior:
|
||||
|
||||
| Parameter | Env Variable | Default | Description |
|
||||
|-----------|---------------------------|---------|------------------------------------------|
|
||||
| --------- | ------------------------- | ------- | ---------------------------------------- |
|
||||
| Fn | `SGLANG_CACHE_DIT_FN` | 1 | Number of first blocks to always compute |
|
||||
| Bn | `SGLANG_CACHE_DIT_BN` | 0 | Number of last blocks to always compute |
|
||||
| W | `SGLANG_CACHE_DIT_WARMUP` | 4 | Warmup steps before caching starts |
|
||||
@@ -918,7 +913,7 @@ DBCache controls block-level caching behavior:
|
||||
TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
| Parameter | Env Variable | Default | Description |
|
||||
|-----------|-------------------------------|---------|---------------------------------|
|
||||
| --------- | ----------------------------- | ------- | ------------------------------- |
|
||||
| Enable | `SGLANG_CACHE_DIT_TAYLORSEER` | false | Enable TaylorSeer calibrator |
|
||||
| Order | `SGLANG_CACHE_DIT_TS_ORDER` | 1 | Taylor expansion order (1 or 2) |
|
||||
|
||||
@@ -950,7 +945,7 @@ which to use cached results.
|
||||
SCM is configured with presets:
|
||||
|
||||
| Preset | Compute Ratio | Speed | Quality |
|
||||
|----------|---------------|----------|------------|
|
||||
| -------- | ------------- | -------- | ---------- |
|
||||
| `none` | 100% | Baseline | Best |
|
||||
| `slow` | ~75% | ~1.3x | High |
|
||||
| `medium` | ~50% | ~2x | Good |
|
||||
@@ -981,7 +976,7 @@ sglang generate --model-path Qwen/Qwen-Image \
|
||||
#### SCM Policy
|
||||
|
||||
| Policy | Env Variable | Description |
|
||||
|-----------|---------------------------------------|---------------------------------------------|
|
||||
| --------- | ------------------------------------- | ------------------------------------------- |
|
||||
| `dynamic` | `SGLANG_CACHE_DIT_SCM_POLICY=dynamic` | Adaptive caching based on content (default) |
|
||||
| `static` | `SGLANG_CACHE_DIT_SCM_POLICY=static` | Fixed caching pattern |
|
||||
|
||||
@@ -990,7 +985,7 @@ sglang generate --model-path Qwen/Qwen-Image \
|
||||
All Cache-DiT parameters can be set via the following environment variables:
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
|-------------------------------------|---------|------------------------------------------|
|
||||
| ----------------------------------- | ------- | ---------------------------------------- |
|
||||
| `SGLANG_CACHE_DIT_ENABLED` | false | Enable Cache-DiT acceleration |
|
||||
| `SGLANG_CACHE_DIT_FN` | 1 | First N blocks to always compute |
|
||||
| `SGLANG_CACHE_DIT_BN` | 0 | Last N blocks to always compute |
|
||||
@@ -1009,7 +1004,7 @@ All Cache-DiT parameters can be set via the following environment variables:
|
||||
SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
|
||||
|
||||
| Model Family | Example Models |
|
||||
|--------------|-------------------------------------------|
|
||||
| ------------ | ------------------------------------ |
|
||||
| Wan | Wan2.1, Wan2.2 |
|
||||
| Flux | FLUX.1-dev, FLUX.2-dev, FLUX.2-Klein |
|
||||
| Z-Image | Z-Image-Turbo |
|
||||
@@ -1075,6 +1070,7 @@ sglang generate \
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `--profile`: Enable profiling for the denoising stage
|
||||
- `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5)
|
||||
- Smaller values reduce trace file size
|
||||
@@ -1094,6 +1090,7 @@ sglang generate \
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising
|
||||
|
||||
### Output Location
|
||||
@@ -1109,15 +1106,16 @@ The exact output file path will be shown in the console output, for example:
|
||||
### View Traces
|
||||
|
||||
Load and visualize trace files at:
|
||||
|
||||
- https://ui.perfetto.dev/ (recommended)
|
||||
- chrome://tracing (Chrome only)
|
||||
|
||||
For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`.
|
||||
|
||||
|
||||
### `--perf-dump-path` (Stage/Step Timing Dump)
|
||||
|
||||
Besides profiler traces, you can also dump a lightweight JSON report that contains:
|
||||
|
||||
- stage-level timing breakdown for the full pipeline
|
||||
- step-level timing breakdown for the denoising stage (per diffusion step)
|
||||
|
||||
@@ -1177,6 +1175,7 @@ nsys profile \
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `--delay N`: Wait N seconds before starting capture (skip initialization overhead)
|
||||
- `--duration N`: Capture for N seconds (focus on specific stages)
|
||||
- `--force-overwrite`: Overwrite existing output files
|
||||
@@ -1202,16 +1201,19 @@ This guide outlines the requirements for contributing to the SGLang Diffusion mo
|
||||
We follow a structured commit message format to maintain a clean history.
|
||||
|
||||
**Format:**
|
||||
|
||||
```text
|
||||
[diffusion] <scope>: <subject>
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
|
||||
- `[diffusion] cli: add --perf-dump-path argument`
|
||||
- `[diffusion] scheduler: fix deadlock in batch processing`
|
||||
- `[diffusion] model: support Stable Diffusion 3.5`
|
||||
|
||||
**Rules:**
|
||||
|
||||
- **Prefix**: Always start with `[diffusion]`.
|
||||
- **Scope** (Optional): `cli`, `scheduler`, `model`, `pipeline`, `docs`, etc.
|
||||
- **Subject**: Imperative mood, short and clear (e.g., "add feature" not "added feature").
|
||||
@@ -1223,11 +1225,13 @@ For PRs that impact **latency**, **throughput**, or **memory usage**, you **shou
|
||||
### How to Generate a Report
|
||||
|
||||
1. **Baseline**: run the benchmark (for a single generation task)
|
||||
|
||||
```bash
|
||||
$ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path baseline.json
|
||||
```
|
||||
|
||||
2. **New**: run the same benchmark, without modifying any server_args or sampling_params
|
||||
|
||||
```bash
|
||||
$ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path new.json
|
||||
```
|
||||
@@ -1260,6 +1264,7 @@ SGLang diffusion uses a modular pipeline architecture built around two key conce
|
||||
- **`PipelineStage`**: Modular components (prompt encoding, denoising loop, VAE decoding, etc.)
|
||||
|
||||
To add a new model, you'll need to define:
|
||||
|
||||
1. **`PipelineConfig`**: Static model configurations (paths, precision settings)
|
||||
2. **`SamplingParams`**: Runtime generation parameters (prompt, guidance_scale, steps)
|
||||
3. **`ComposedPipeline`**: Chain together pipeline stages
|
||||
@@ -0,0 +1,9 @@
|
||||
Image Generation
|
||||
================
|
||||
|
||||
Models for generating images and videos using diffusion.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
diffusion_models.md
|
||||
@@ -0,0 +1,14 @@
|
||||
Supported Models
|
||||
================
|
||||
|
||||
SGLang supports a wide variety of model architectures for different use cases.
|
||||
Browse by category below to find models suited for your needs.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
text_generation/index
|
||||
image_generation/index
|
||||
retrieval_ranking/index
|
||||
specialized/index
|
||||
extending/index
|
||||
@@ -0,0 +1,11 @@
|
||||
Retrieval & Ranking
|
||||
===================
|
||||
|
||||
Models for embeddings, reranking, and classification.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
embedding_models.md
|
||||
rerank_models.md
|
||||
classify_models.md
|
||||
@@ -0,0 +1,9 @@
|
||||
Specialized Models
|
||||
==================
|
||||
|
||||
Models for specialized tasks like reward modeling.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
reward_models.md
|
||||
+4
-1
@@ -1,5 +1,7 @@
|
||||
# Diffusion Language Models
|
||||
|
||||
> This page covers **text generation** using diffusion-based LLMs. For **image and video generation**, see [Diffusion Models](../image_generation/diffusion_models.md).
|
||||
|
||||
Diffusion language models have shown promise for non-autoregressive text generation with parallel decoding capabilities. Unlike auto-regressive language models, different diffusion language models require different decoding strategies.
|
||||
|
||||
## Example Launch Command
|
||||
@@ -25,6 +27,7 @@ threshold: 0.95
|
||||
# Default: 32, for LLaDA2MoeModelLM
|
||||
block_size: 32
|
||||
```
|
||||
|
||||
## Example Client Code Snippet
|
||||
|
||||
Just like other supported models, diffusion language models can be used via the REST API or Python client.
|
||||
@@ -79,5 +82,5 @@ curl -X POST "http://127.0.0.1:30000/generate" \
|
||||
Below the supported models are summarized in a table.
|
||||
|
||||
| Model Family | Example Model | Description |
|
||||
| ------------------------------------------ | -------------------------------------- | --------------------------------------------------------------------------- |
|
||||
| -------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------- |
|
||||
| **LLaDA2.0 (mini, flash)** | `inclusionAI/LLaDA2.0-flash` | LLaDA2.0-flash is a diffusion language model featuring a 100B Mixture-of-Experts (MoE) architecture. |
|
||||
+2
@@ -66,3 +66,5 @@ in the GitHub search bar.
|
||||
| **StarCoder2** (3B-15B) | `bigcode/starcoder2-7b` | StarCoder2 is a family of open large language models (LLMs) specialized for code generation and understanding. It is the successor to StarCoder, jointly developed by the BigCode project (a collaboration between Hugging Face, ServiceNow Research, and other contributors). |
|
||||
| **Jet-Nemotron** | `jet-ai/Jet-Nemotron-2B` | Jet-Nemotron is a new family of hybrid-architecture language models that surpass state-of-the-art open-source full-attention language models, while achieving significant efficiency gains. |
|
||||
| **Trinity** (Nano, Mini) | `arcee-ai/Trinity-Mini` | Arcee's foundational MoE Trinity family of models, open weights under Apache 2.0. |
|
||||
| **Falcon-H1** (0.5B–34B) | `tiiuae/Falcon-H1-34B-Instruct` | TII's hybrid Mamba-Transformer architecture combining attention and state-space models for efficient long-context inference. |
|
||||
| **Hunyuan-Large** (389B, MoE) | `tencent/Tencent-Hunyuan-Large` | Tencent's open-source MoE model with 389B total / 52B active parameters, featuring Cross-Layer Attention (CLA) for improved efficiency. |
|
||||
@@ -0,0 +1,11 @@
|
||||
Text Generation
|
||||
===============
|
||||
|
||||
Models for generating text from text or multimodal inputs.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
generative_models.md
|
||||
multimodal_language_models.md
|
||||
diffusion_language_models.md
|
||||
+2
@@ -50,6 +50,8 @@ in the GitHub search bar.
|
||||
| **NVIDIA Nemotron Nano 2.0 VL** | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16` | NVIDIA Nemotron Nano v2 VL enables multi-image reasoning and video understanding, along with strong document intelligence, visual Q&A and summarization capabilities. It builds on Nemotron Nano V2, a hybrid Mamba-Transformer LLM, in order to achieve higher inference throughput in long document and video scenarios. | Use `--trust-remote-code`. You may need to adjust `--max-mamba-cache-size` [default is 512] to fit memory constraints. |
|
||||
| **Ernie4.5-VL** | `baidu/ERNIE-4.5-VL-28B-A3B-PT` | Baidu's vision-language models(28B,424B). Support image and video comprehension, and also support thinking. | |
|
||||
| **JetVLM** | | JetVLM is an vision-language model designed for high-performance multimodal understanding and generation tasks built upon Jet-Nemotron. | Coming soon |
|
||||
| **Step3-VL** (10B) | `stepfun-ai/Step3-VL-10B` | StepFun's lightweight open-source 10B parameter VLM for multimodal intelligence, excelling in visual perception, complex reasoning, and human alignment. | |
|
||||
| **Qwen3-Omni** | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | Alibaba's omni-modal MoE model. Currently supports the **Thinker** component (multimodal understanding for text, images, audio, and video), while the **Talker** component (audio generation) is not yet supported. | |
|
||||
|
||||
## Video Input Support
|
||||
|
||||
Reference in New Issue
Block a user