[Docs] Add Falcon H1, Hunyuan-Large, Qwen3-Omni support and update Diffusion usage (#17888)

Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: Ratish P <114130421+Ratish1@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
2026-02-06 13:17:51 -08:00
parent f2e0048d06
commit c850a8a41a
25 changed files with 664 additions and 558 deletions
@@ -0,0 +1,19 @@
 # Diffusion
 SGLang supports two categories of diffusion models for different use cases. This page covers image and video generation; for diffusion LLMs, see [Diffusion LLMs](diffusion_llms.md).
 ## Image & Video Generation Models
 For generating images and videos from text prompts, SGLang supports [many](../supported_models/image_generation/diffusion_models.md#image-generation-models) models like:
 - **FLUX, Qwen-Image** - High-quality image generation
 - **Wan 2.2, HunyuanVideo** - Video generation
 ```bash
 # Example: Launch FLUX for image generation
 python3 -m sglang.launch_server \
  --model-path black-forest-labs/FLUX.2-klein-4B \
  --host 0.0.0.0 --port 30000
 ```
 **Full model list:** [Diffusion Models](../supported_models/image_generation/diffusion_models.md)
@@ -0,0 +1,14 @@
 # Diffusion Language Models (dLLMs)
 These are text-generation models that use diffusion (denoising) instead of autoregressive decoding:
 - **LLaDA** - Large Language Diffusion with mAsking
 ```bash
 # Example: Launch LLaDA for text generation
 python3 -m sglang.launch_server \
  --model-path GSAI-ML/LLaDA-8B-Instruct \
  --host 0.0.0.0 --port 30000
 ```
 **Full model list:** [Diffusion Language Models](../supported_models/text_generation/diffusion_language_models.md)
@@ -9,7 +9,7 @@
    "SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).\n",
    "\n",
-    "This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported_models/embedding_models.md)\n"
+    "This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported_models/retrieval_ranking/embedding_models.md)\n"
   ]
  },
  {
@@ -173,7 +173,7 @@
   "metadata": {},
   "source": [
    "## Multi-Modal Embedding Model\n",
-    "Please refer to [Multi-Modal Embedding Model](../supported_models/embedding_models.md)"
+    "Please refer to [Multi-Modal Embedding Model](../supported_models/retrieval_ranking/embedding_models.md)"
   ]
  }
 ],
@@ -10,7 +10,7 @@
    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).\n",
    "This tutorial covers the vision APIs for vision language models.\n",
    "\n",
-    "SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported_models/multimodal_language_models.md).\n",
+    "SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported_models/text_generation/multimodal_language_models.md).\n",
    "\n",
    "As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py)."
   ]
@@ -35,6 +35,8 @@ Its core features include:
   basic_usage/native_api.ipynb
   basic_usage/sampling_params.md
   basic_usage/popular_model_usage.rst
   basic_usage/diffusion_llms.md
   basic_usage/diffusion.md
 .. toctree::
   :maxdepth: 1
@@ -67,21 +69,14 @@ Its core features include:
   advanced_features/sglang_for_rl.md
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
   :caption: Supported Models
-   supported_models/generative_models.md
+   supported_models/text_generation/index
-   supported_models/multimodal_language_models.md
+   supported_models/image_generation/index
-   supported_models/diffusion_language_models.md
+   supported_models/retrieval_ranking/index
-   supported_models/diffusion_models.md
+   supported_models/specialized/index
-   supported_models/embedding_models.md
+   supported_models/extending/index
   supported_models/reward_models.md
   supported_models/rerank_models.md
   supported_models/classify_models.md
   supported_models/support_new_models.md
   supported_models/transformers_fallback.md
   supported_models/modelscope.md
   supported_models/mindspore_models.md
 .. toctree::
   :maxdepth: 1
@@ -8,5 +8,6 @@ Ascend NPUs
   ascend_npu_support_models.md
   ascend_npu_deepseek_example.md
   ascend_npu_qwen3_examples.md
   mindspore_backend.md
   ascend_contribution_guide.md
   ascend_npu_best_practice.md
@@ -6,8 +6,8 @@ MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc
 ## Requirements
-MindSpore currently only supports Ascend NPU devices. Users need to first install CANN 8.5.
+MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages.
-The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com).
+The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2.
 ## Supported Models
@@ -19,7 +19,7 @@ Currently, the following models are supported:
 ## Installation
-> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](../platforms/ascend_npu.md) and then install `sgl-mindspore`:
+> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](ascend_npu.md) and then install `sgl-mindspore`:
 ```shell
 git clone https://github.com/mindspore-lab/sgl-mindspore.git
@@ -32,9 +32,9 @@ pip install -e .
 Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example.
-### Offline inference
+### Offline infer
-Use the following script for offline inference:
+Use the following script for offline infer:
 ```python
 import sglang as sgl
@@ -385,7 +385,7 @@
    "## Multi-modal Generation\n",
    "\n",
    "You may use SGLang frontend language to define multi-modal prompts.\n",
-    "See [here](https://docs.sglang.io/supported_models/generative_models.html) for supported models."
+    "See [here](https://docs.sglang.io/supported_models/text_generation/generative_models.html) for supported models."
   ]
  },
  {
@@ -0,0 +1,11 @@
 Extending SGLang
 ================
 Adding new models and alternative backends.
 .. toctree::
   :maxdepth: 1
   support_new_models.md
   transformers_fallback.md
   modelscope.md
@@ -1,320 +1,320 @@
-# How to Support New Models
+# How to Support New Models
-
+
-This document explains how to add support for new language models and multimodal large language models (MLLMs) in
+This document explains how to add support for new language models and multimodal large language models (MLLMs) in
-SGLang. It also covers how to test new models and register external implementations.
+SGLang. It also covers how to test new models and register external implementations.
-
+
-## How to Support a New Language Model
+## How to Support a New Language Model
-
+
-To support a new model in SGLang, you only need to add a single file under
+To support a new model in SGLang, you only need to add a single file under
-the [SGLang Models Directory](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models). You can learn
+the [SGLang Models Directory](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models). You can learn
-from existing model implementations and create a new file for your model. For most models, you should be able to find a
+from existing model implementations and create a new file for your model. For most models, you should be able to find a
-similar model to start with (e.g., starting from Llama). Also refer how
+similar model to start with (e.g., starting from Llama). Also refer how
-to [port a Model from vLLM to SGLang](#port-a-model-from-vllm-to-sglang)
+to [port a Model from vLLM to SGLang](#port-a-model-from-vllm-to-sglang)
-
+
-## How to Support a New Multimodal Large Language Model
+## How to Support a New Multimodal Large Language Model
-
+
-To support a new multimodal large language model (MLLM) in SGLang, there are several key components in addition to the
+To support a new multimodal large language model (MLLM) in SGLang, there are several key components in addition to the
-standard LLM support:
+standard LLM support:
-
+
-1. **Register your new model as multimodal**:
+1. **Register your new model as multimodal**:
-   Extend `is_multimodal_model`
+   Extend `is_multimodal_model`
-   in [model_config.py](https://github.com/sgl-project/sglang/blob/0ab3f437aba729b348a683ab32b35b214456efc7/python/sglang/srt/configs/model_config.py#L561)
+   in [model_config.py](https://github.com/sgl-project/sglang/blob/0ab3f437aba729b348a683ab32b35b214456efc7/python/sglang/srt/configs/model_config.py#L561)
-   to return `True` for your model.
+   to return `True` for your model.
-
+
-2. **Register a new chat-template**:
+2. **Register a new chat-template**:
-   Only when your default chat-template is unable to accept images as input: Register a new chat template in [conversation.py](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/conversation.py) and the corresponding matching function.
+   Only when your default chat-template is unable to accept images as input: Register a new chat template in [conversation.py](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/conversation.py) and the corresponding matching function.
-
+
-3. **Multimodal Data Processor**:
+3. **Multimodal Data Processor**:
-   Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your
+   Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your
-   model’s dedicated processor.
+   model’s dedicated processor.
-   See [multimodal_processor.py](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors)
+   See [multimodal_processor.py](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors)
-   for more details.
+   for more details.
-
+
-4. **Handle Multimodal Tokens**:
+4. **Handle Multimodal Tokens**:
-   Implement a `pad_input_ids` function for your new model. In this function, multimodal tokens in the prompt should be
+   Implement a `pad_input_ids` function for your new model. In this function, multimodal tokens in the prompt should be
-   expanded (if necessary) and padded with multimodal-data-hashes so that SGLang can recognize different multimodal data
+   expanded (if necessary) and padded with multimodal-data-hashes so that SGLang can recognize different multimodal data
-   with `RadixAttention`.
+   with `RadixAttention`.
-
+
-5. **Handle Image Feature Extraction**:
+5. **Handle Image Feature Extraction**:
-   Implement a `get_image_feature` function for your new model, which extracts image features from raw image data and converts them into the embeddings used by the language model.
+   Implement a `get_image_feature` function for your new model, which extracts image features from raw image data and converts them into the embeddings used by the language model.
-
+
-6. **Adapt to Vision Attention**:
+6. **Adapt to Vision Attention**:
-   Adapt the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.
+   Adapt the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.
-
+
-You can refer to [Qwen2VL](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen2_vl.py) or
+You can refer to [Qwen2VL](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen2_vl.py) or
-other mllm implementations. These models demonstrate how to correctly handle both multimodal and textual inputs.
+other mllm implementations. These models demonstrate how to correctly handle both multimodal and textual inputs.
-
+
-## Testing and Debugging
+## Testing and Debugging
-
+
-Please note all your testing and benchmarking results in PR description.
+Please note all your testing and benchmarking results in PR description.
-
+
-### Interactive Debugging
+### Interactive Debugging
-
+
-For interactive debugging, compare the outputs of Hugging Face/Transformers and SGLang. The following two commands
+For interactive debugging, compare the outputs of Hugging Face/Transformers and SGLang. The following two commands
-should give the same text output and very similar prefill logits:
+should give the same text output and very similar prefill logits:
-
+
- Get the reference output:
+- Get the reference output:
-  ```bash
+  ```bash
-  python3 scripts/playground/reference_hf.py --model-path [new model] --model-type {text,mllm}
+  python3 scripts/playground/reference_hf.py --model-path [new model] --model-type {text,mllm}
-  ```
+  ```
- Get the SGLang output:
+- Get the SGLang output:
-  ```bash
+  ```bash
-  python3 -m sglang.bench_one_batch --correct --model [new model]
+  python3 -m sglang.bench_one_batch --correct --model [new model]
-  ```
+  ```
-
+
-### Add the Model to the Test Suite
+### Add the Model to the Test Suite
-
+
-To ensure the new model is well maintained, add it to the test suite by including it in the `ALL_OTHER_MODELS` list in
+To ensure the new model is well maintained, add it to the test suite by including it in the `ALL_OTHER_MODELS` list in
-the [test_generation_models.py](https://github.com/sgl-project/sglang/blob/main/test/srt/models/test_generation_models.py)
+the [test_generation_models.py](https://github.com/sgl-project/sglang/blob/main/test/srt/models/test_generation_models.py)
-file, test the new model on your local machine and report the results on demonstrative benchmarks (GSM8K, MMLU, MMMU,
+file, test the new model on your local machine and report the results on demonstrative benchmarks (GSM8K, MMLU, MMMU,
-MMMU-Pro, etc.) in your PR. \\
+MMMU-Pro, etc.) in your PR. \\
-For VLMs, also include a test in `test_vision_openai_server_{x}.py` (e.g. [test_vision_openai_server_a.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server_a.py), [test_vision_openai_server_b.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server_b.py)).
+For VLMs, also include a test in `test_vision_openai_server_{x}.py` (e.g. [test_vision_openai_server_a.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server_a.py), [test_vision_openai_server_b.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server_b.py)).
-
+
-
+This is an example command to run to test a new model on your local machine:
-This is an example command to run to test a new model on your local machine:
+
-
+```bash
-```bash
+ONLY_RUN=Qwen/Qwen2-1.5B python3 -m unittest test_generation_models.TestGenerationModels.test_others
-ONLY_RUN=Qwen/Qwen2-1.5B python3 -m unittest test_generation_models.TestGenerationModels.test_others
+```
-```
+
-
+### Benchmark
-### Benchmark
+
-
+- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow https://docs.sglang.io/developer_guide/benchmark_and_profiling.html to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer).
- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow https://docs.sglang.io/developer_guide/benchmark_and_profiling.html to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer).
+- **(Optional) Other evals**: If you ran other evals, please note the results in PR description.
- **(Optional) Other evals**: If you ran other evals, please note the results in PR description.
+
-
+## Port a Model from vLLM to SGLang
-## Port a Model from vLLM to SGLang
+
-
+The [vLLM Models Directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) is a valuable
-The [vLLM Models Directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) is a valuable
+resource, as vLLM covers many models. SGLang reuses vLLM’s interface and some layers, making it easier to port models
-resource, as vLLM covers many models. SGLang reuses vLLM’s interface and some layers, making it easier to port models
+from vLLM to SGLang.
-from vLLM to SGLang.
+
-
+To port a model from vLLM to SGLang:
-To port a model from vLLM to SGLang:
+
-
+- Compare these two files for guidance:
- Compare these two files for guidance:
+  - [SGLang Llama Implementation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama.py)
-    - [SGLang Llama Implementation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama.py)
+  - [vLLM Llama Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py)
-    - [vLLM Llama Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py)
+- The major differences include:
- The major differences include:
+  - **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`).
-    - **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`).
+  - **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.**
-    - **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.**
+  - **Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.**
-    - **Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.**
+  - **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers.
-    - **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers.
+  - **Remove `Sample`.**
-    - **Remove `Sample`.**
+  - **Change the `forward()` functions** and add a `forward_batch()` method.
-    - **Change the `forward()` functions** and add a `forward_batch()` method.
+  - **Add `EntryClass`** at the end.
-    - **Add `EntryClass`** at the end.
+  - **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components.
-    - **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components.
+
-
+Note: make sure you add your new model to the supported models list in the supported models documentation.
-Note: make sure you add your new model to the supported models list in the supported models documentation.
+
-
+## Registering an External Model Implementation
-## Registering an External Model Implementation
+
-
+In addition to the methods above, you can register your new model with the `ModelRegistry` before launching the server.
-In addition to the methods above, you can register your new model with the `ModelRegistry` before launching the server.
+This allows you to integrate your model without modifying the source code.
-This allows you to integrate your model without modifying the source code.
+
-
+For example:
-For example:
+
-
+```python
-```python
+from sglang.srt.models.registry import ModelRegistry
-from sglang.srt.models.registry import ModelRegistry
+from sglang.srt.entrypoints.http_server import launch_server
-from sglang.srt.entrypoints.http_server import launch_server
+
-
+# For a single model, add it to the registry:
-# For a single model, add it to the registry:
+ModelRegistry.models[model_name] = model_class
-ModelRegistry.models[model_name] = model_class
+
-
+# For multiple models, you can imitate the import_model_classes() function:
-# For multiple models, you can imitate the import_model_classes() function:
+from functools import lru_cache
-from functools import lru_cache
+
-
+@lru_cache()
-@lru_cache()
+def import_new_model_classes():
-def import_new_model_classes():
+    model_arch_name_to_cls = {}
-    model_arch_name_to_cls = {}
+    # Populate model_arch_name_to_cls with your new model classes.
-    # Populate model_arch_name_to_cls with your new model classes.
+    ...
-    ...
+    return model_arch_name_to_cls
-    return model_arch_name_to_cls
+
-
+ModelRegistry.models.update(import_new_model_classes())
-ModelRegistry.models.update(import_new_model_classes())
+
-
+# Launch the server with your server arguments:
-# Launch the server with your server arguments:
+launch_server(server_args)
-launch_server(server_args)
+```
-```
+
-
+## Example: Implementing and Serving a Llama Wrapper Model
-## Example: Implementing and Serving a Llama Wrapper Model
+
-
+Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](https://github.com/sgl-project/sglang/blob/main/docs/basic_usage/offline_engine_api.ipynb).
-Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](https://github.com/sgl-project/sglang/blob/main/docs/basic_usage/offline_engine_api.ipynb).
+
-
+### Implementing Our Model
-### Implementing Our Model
+
-
+To keep things simple, this new model will be a simple wrapper around [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), and our goal will be just to bias the output logits for each `forward` call by taking the square root of each individual logit.
-To keep things simple, this new model will be a simple wrapper around [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), and our goal will be just to bias the output logits for each `forward` call by taking the square root of each individual logit.
+
-
+Let's start by defining our model in a file called `llama_wrapper.py`.
-Let's start by defining our model in a file called `llama_wrapper.py`.
+The first step is to import the necessary libraries from SRT, which is SGLang's internal backend.
-The first step is to import the necessary libraries from SRT, which is SGLang's internal backend.
+
-
+```python
-```python
+# In the file `llama_wrapper.py`
-# In the file `llama_wrapper.py`
+
-
+import torch
-import torch
+from transformers import LlamaConfig
-from transformers import LlamaConfig
+from typing import Optional
-from typing import Optional
+from sglang.srt.layers.logits_processor import LogitsProcessorOutput
-from sglang.srt.layers.logits_processor import LogitsProcessorOutput
+from sglang.srt.layers.quantization.base_config import QuantizationConfig
-from sglang.srt.layers.quantization.base_config import QuantizationConfig
+from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors
-from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors
+
-
+from sglang.srt.models.llama import LlamaForCausalLM
-from sglang.srt.models.llama import LlamaForCausalLM
+```
-```
+
-
+Next, we declare a new `class` for our model and have it inherit from `LlamaForCausalLM`, which allows our model to access `LlamaForCausalLM`'s predefined modules and layers, such as `LlamaAttention` and `LlamaMLP`.
-Next, we declare a new `class` for our model and have it inherit from `LlamaForCausalLM`, which allows our model to access `LlamaForCausalLM`'s predefined modules and layers, such as `LlamaAttention` and `LlamaMLP`.
+Note that almost all model implementations take in `config` and `quant_config` as arguments for their `__init__` method; `config` and `quant_config` are passed in via [`model_loader/loader.py`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_loader/loader.py#L219).
-Note that almost all model implementations take in `config` and `quant_config` as arguments for their `__init__` method; `config` and `quant_config` are passed in via [`model_loader/loader.py`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_loader/loader.py#L219).
+Because we have inherited from `LlamaForCausalLM`, we can pass our parameters directly to its constructor, which will set the member variables for us.
-Because we have inherited from `LlamaForCausalLM`, we can pass our parameters directly to its constructor, which will set the member variables for us.
+
-
+```python
-```python
+class LlamaWrapper(LlamaForCausalLM):
-class LlamaWrapper(LlamaForCausalLM):
+    def __init__(
-    def __init__(
+        self,
-        self,
+        config: LlamaConfig,
-        config: LlamaConfig,
+        quant_config: Optional[QuantizationConfig] = None,
-        quant_config: Optional[QuantizationConfig] = None,
+        prefix: str = "",
-        prefix: str = "",
+    ) -> None:
-    ) -> None:
+        super().__init__(config=config, quant_config=quant_config, prefix=prefix)
-        super().__init__(config=config, quant_config=quant_config, prefix=prefix)
+```
-```
+
-
+Now, we want to define the `forward` method, which is what will be called at inference time.
-Now, we want to define the `forward` method, which is what will be called at inference time.
+Note that the signature for `forward` is essentially the same for any model; you can take a look at the other models defined in the [`models` directory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/) for references.
-Note that the signature for `forward` is essentially the same for any model; you can take a look at the other models defined in the [`models` directory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/) for references.
+To see where exactly `forward` is called in the SGLang runtime's internals, take a look at [`forward_decode`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1705) and [`forward_extend`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1724) in the [`ModelRunner` class](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/model_runner.py).
-To see where exactly `forward` is called in the SGLang runtime's internals, take a look at [`forward_decode`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1705) and [`forward_extend`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1724) in the [`ModelRunner` class](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/model_runner.py).
+
-
+```python
-```python
+    @torch.no_grad()
-    @torch.no_grad()
+    def forward(
-    def forward(
+        self,
-        self,
+        input_ids: torch.Tensor,
-        input_ids: torch.Tensor,
+        positions: torch.Tensor,
-        positions: torch.Tensor,
+        forward_batch: ForwardBatch,
-        forward_batch: ForwardBatch,
+        pp_proxy_tensors: Optional[PPProxyTensors] = None,
-        pp_proxy_tensors: Optional[PPProxyTensors] = None,
+        input_embeds: Optional[torch.Tensor] = None,
-        input_embeds: Optional[torch.Tensor] = None,
+        get_embedding: bool = False,
-        get_embedding: bool = False,
+    ) -> LogitsProcessorOutput:
-    ) -> LogitsProcessorOutput:
+```
-```
+
-
+We now call the `__call__` method for `self.model` (which is a member variable that `LlamaForCausalLM` defines in its `__init__` method), which eventually calls `LlamaForCausalLM`'s `forward` method.
-We now call the `__call__` method for `self.model` (which is a member variable that `LlamaForCausalLM` defines in its `__init__` method), which eventually calls `LlamaForCausalLM`'s `forward` method.
+After that, we feed the `hidden_states` into our model's `LogitsProcessor` (again defined in `LlamaForCausalLM`).
-After that, we feed the `hidden_states` into our model's `LogitsProcessor` (again defined in `LlamaForCausalLM`).
+
-
+```python
-```python
+        hidden_states = self.model(
-        hidden_states = self.model(
+            input_ids,
-            input_ids,
+            positions,
-            positions,
+            forward_batch,
-            forward_batch,
+            input_embeds,
-            input_embeds,
+            pp_proxy_tensors=pp_proxy_tensors,
-            pp_proxy_tensors=pp_proxy_tensors,
+        )
-        )
+
-
+        res: LogitsProcessorOutput = self.logits_processor(
-        res: LogitsProcessorOutput = self.logits_processor(
+            input_ids,
-            input_ids,
+            hidden_states,
-            hidden_states,
+            self.lm_head,
-            self.lm_head,
+            forward_batch,
-            forward_batch,
+        )
-        )
+```
-```
+
-
+After receiving the logits for the next token, we can finally perform our biasing step.
-After receiving the logits for the next token, we can finally perform our biasing step.
+
-
+```python
-```python
+        orig_logits = res.next_token_logits
-        orig_logits = res.next_token_logits
+        res.next_token_logits = torch.where(
-        res.next_token_logits = torch.where(
+            orig_logits > 0,
-            orig_logits > 0,
+            orig_logits.sqrt(),
-            orig_logits.sqrt(),
+            orig_logits
-            orig_logits
+        )
-        )
+
-
+        return res
-        return res
+```
-```
+
-Now, our `LlamaWrapper` model is created and ready to be served!
+Now, our `LlamaWrapper` model is created and ready to be served!
-
+
-### Serving Our Model Via SGLang's Offline Engine
+### Serving Our Model Via SGLang's Offline Engine
-
+
-The next step of this walkthrough involves hosting our new model offline, so that it can be served locally and without an HTTP server.
+The next step of this walkthrough involves hosting our new model offline, so that it can be served locally and without an HTTP server.
-
+
-First, create a new file called `run.py`.
+First, create a new file called `run.py`.
-Now, we must ensure that SGLang's `ModelRegistry` can find our model.
+Now, we must ensure that SGLang's `ModelRegistry` can find our model.
-To do this, we first download the model's configuration and weights from Huggingface.
+To do this, we first download the model's configuration and weights from Huggingface.
-
+
-```python
+```python
-# In the file `run.py`
+# In the file `run.py`
-
+
-import asyncio
+import asyncio
-from functools import lru_cache
+from functools import lru_cache
-from huggingface_hub import snapshot_download
+from huggingface_hub import snapshot_download
-from llama_wrapper import LlamaWrapper # Make sure to import our new model!
+from llama_wrapper import LlamaWrapper # Make sure to import our new model!
-import sglang as sgl
+import sglang as sgl
-from sglang.srt.models.registry import ModelRegistry
+from sglang.srt.models.registry import ModelRegistry
-
+
-# Make sure to request access to this model on Huggingface, then export your
+# Make sure to request access to this model on Huggingface, then export your
-# `HF_TOKEN` to download the model snapshot
+# `HF_TOKEN` to download the model snapshot
-llama_dir = snapshot_download(
+llama_dir = snapshot_download(
-    repo_id="meta-llama/Llama-3.1-8B-Instruct",
+    repo_id="meta-llama/Llama-3.1-8B-Instruct",
-    local_dir="./llama_ckpt",
+    local_dir="./llama_ckpt",
-)
+)
-```
+```
-
+
-Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`.
+Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`.
-That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model.
+That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model.
-
+
-```python
+```python
-{
+{
-  "architectures": [
+  "architectures": [
-   #  "LlamaForCausalLM"
+   #  "LlamaForCausalLM"
-    "LlamaWrapper"
+    "LlamaWrapper"
-  ],
+  ],
-  ...
+  ...
-}
+}
-```
+```
-
+
-However, if we don't link our `LlamaWrapper` class to the "LlamaWrapper" registry keyword, then SGLang won't be able to find our model.
+However, if we don't link our `LlamaWrapper` class to the "LlamaWrapper" registry keyword, then SGLang won't be able to find our model.
-Thus, to register our `LlamaWrapper`, we want to follow the steps in the above section titled "Registering an External Model Implementation".
+Thus, to register our `LlamaWrapper`, we want to follow the steps in the above section titled "Registering an External Model Implementation".
-
+
-```python
+```python
-@lru_cache()
+@lru_cache()
-def import_new_model_classes():
+def import_new_model_classes():
-    model_arch_name_to_cls = {"LlamaWrapper": LlamaWrapper}
+    model_arch_name_to_cls = {"LlamaWrapper": LlamaWrapper}
-    return model_arch_name_to_cls
+    return model_arch_name_to_cls
-
+
-ModelRegistry.models.update(import_new_model_classes())
+ModelRegistry.models.update(import_new_model_classes())
-```
+```
-
+
-Lastly, when we create our `Engine`, we just pass in the path to the local model directory.
+Lastly, when we create our `Engine`, we just pass in the path to the local model directory.
-Then, our `LlamaWrapper` is ready to be served; for this walkthrough, we will use SGLang `Engine`'s non-streaming asynchronous generation endpoint.
+Then, our `LlamaWrapper` is ready to be served; for this walkthrough, we will use SGLang `Engine`'s non-streaming asynchronous generation endpoint.
-
+
-```python
+```python
-def main():
+def main():
-    llm = sgl.Engine(model_path="./llama_ckpt")
+    llm = sgl.Engine(model_path="./llama_ckpt")
-    sampling_params = {"temperature": 0.2, "top_k": 5}
+    sampling_params = {"temperature": 0.2, "top_k": 5}
-    prompts = [
+    prompts = [
-        "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
+        "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
-        "Provide a concise factual statement about France’s capital city. The capital of France is",
+        "Provide a concise factual statement about France’s capital city. The capital of France is",
-        "Explain possible future trends in artificial intelligence. The future of AI is",
+        "Explain possible future trends in artificial intelligence. The future of AI is",
-    ]
+    ]
-
+
-    asyncio.run(run_llm(llm, sampling_params, prompts))
+    asyncio.run(run_llm(llm, sampling_params, prompts))
-
+
-    llm.shutdown()
+    llm.shutdown()
-
+
-async def run_llm(
+async def run_llm(
-    llm,
+    llm,
-    sampling_params,
+    sampling_params,
-    prompts,
+    prompts,
-) -> None:
+) -> None:
-    outputs = await llm.async_generate(prompts, sampling_params)
+    outputs = await llm.async_generate(prompts, sampling_params)
-
+
-    for prompt, output in zip(prompts, outputs):
+    for prompt, output in zip(prompts, outputs):
-        print(f"\nPrompt: {prompt}")
+        print(f"\nPrompt: {prompt}")
-        print(f"Generated text: {output['text']}")
+        print(f"Generated text: {output['text']}")
-
+
-if __name__ == "__main__":
+if __name__ == "__main__":
-    main()
+    main()
-```
+```
-
+
-Now, when we call `python run.py`, we will get the outputs of our newly created model!
+Now, when we call `python run.py`, we will get the outputs of our newly created model!
-
+
-
+## Documentation
-## Documentation
+
-Add to table of supported models in [generative_models.md](https://github.com/sgl-project/sglang/blob/main/docs/supported_models/generative_models.md) or [multimodal_language_models.md](https://github.com/sgl-project/sglang/blob/main/docs/supported_models/multimodal_language_models.md)
+Add to table of supported models in [generative_models.md](../text_generation/generative_models.md) or [multimodal_language_models.md](../text_generation/multimodal_language_models.md)
-
+
---
+---
-
+
-By following these guidelines, you can add support for new language models and multimodal large language models in
+By following these guidelines, you can add support for new language models and multimodal large language models in
-SGLang and ensure they are thoroughly tested and easily integrated into the system.
+SGLang and ensure they are thoroughly tested and easily integrated into the system.
@@ -1,5 +1,7 @@
 # Diffusion Models
 > This page covers **image and video generation**. For **text generation** using diffusion LLMs (e.g., LLaDA2.0), see [Diffusion Language Models](../text_generation/diffusion_language_models.md).
 SGLang Diffusion is an inference framework for accelerated image and video generation using diffusion models. It provides an end-to-end unified pipeline with optimized kernels from sgl-kernel and an efficient scheduler loop.
 ## Key Features
@@ -93,33 +95,33 @@ default parameters when initializing and generating videos.
 ### Video Generation Models
 | Model Name                   | Hugging Face Model ID                             | Resolutions         | TeaCache | Sliding Tile Attn | Sage Attn | Video Sparse Attention (VSA) |
-|:-----------------------------|:--------------------------------------------------|:--------------------|:--------:|:-----------------:|:---------:|:----------------------------:|
+| :--------------------------- | :------------------------------------------------ | :------------------ | :------: | :---------------: | :-------: | :--------------------------: |
-| FastWan2.1 T2V 1.3B          | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers`         | 480p                |    ⭕     |         ⭕         |     ⭕     |              ✅               |
+| FastWan2.1 T2V 1.3B          | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers`         | 480p                |    ⭕    |        ⭕         |    ⭕     |              ✅              |
-| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720p                |    ⭕     |         ⭕         |     ⭕     |              ✅               |
+| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720p                |    ⭕    |        ⭕         |    ⭕     |              ✅              |
-| Wan2.2 TI2V 5B               | `Wan-AI/Wan2.2-TI2V-5B-Diffusers`                 | 720p                |    ⭕     |         ⭕         |     ✅     |              ⭕               |
+| Wan2.2 TI2V 5B               | `Wan-AI/Wan2.2-TI2V-5B-Diffusers`                 | 720p                |    ⭕    |        ⭕         |    ✅     |              ⭕              |
-| Wan2.2 T2V A14B              | `Wan-AI/Wan2.2-T2V-A14B-Diffusers`                | 480p<br>720p        |    ❌     |         ❌         |     ✅     |              ⭕               |
+| Wan2.2 T2V A14B              | `Wan-AI/Wan2.2-T2V-A14B-Diffusers`                | 480p<br>720p        |    ❌    |        ❌         |    ✅     |              ⭕              |
-| Wan2.2 I2V A14B              | `Wan-AI/Wan2.2-I2V-A14B-Diffusers`                | 480p<br>720p        |    ❌     |         ❌         |     ✅     |              ⭕               |
+| Wan2.2 I2V A14B              | `Wan-AI/Wan2.2-I2V-A14B-Diffusers`                | 480p<br>720p        |    ❌    |        ❌         |    ✅     |              ⭕              |
-| HunyuanVideo                 | `hunyuanvideo-community/HunyuanVideo`             | 720×1280<br>544×960 |    ❌     |         ✅         |     ✅     |              ⭕               |
+| HunyuanVideo                 | `hunyuanvideo-community/HunyuanVideo`             | 720×1280<br>544×960 |    ❌    |        ✅         |    ✅     |              ⭕              |
-| FastHunyuan                  | `FastVideo/FastHunyuan-diffusers`                 | 720×1280<br>544×960 |    ❌     |         ✅         |     ✅     |              ⭕               |
+| FastHunyuan                  | `FastVideo/FastHunyuan-diffusers`                 | 720×1280<br>544×960 |    ❌    |        ✅         |    ✅     |              ⭕              |
-| Wan2.1 T2V 1.3B              | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`                | 480p                |    ✅     |         ✅         |     ✅     |              ⭕               |
+| Wan2.1 T2V 1.3B              | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`                | 480p                |    ✅    |        ✅         |    ✅     |              ⭕              |
-| Wan2.1 T2V 14B               | `Wan-AI/Wan2.1-T2V-14B-Diffusers`                 | 480p, 720p          |    ✅     |         ✅         |     ✅     |              ⭕               |
+| Wan2.1 T2V 14B               | `Wan-AI/Wan2.1-T2V-14B-Diffusers`                 | 480p, 720p          |    ✅    |        ✅         |    ✅     |              ⭕              |
-| Wan2.1 I2V 480P              | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers`            | 480p                |    ✅     |         ✅         |     ✅     |              ⭕               |
+| Wan2.1 I2V 480P              | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers`            | 480p                |    ✅    |        ✅         |    ✅     |              ⭕              |
-| Wan2.1 I2V 720P              | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers`            | 720p                |    ✅     |         ✅         |     ✅     |              ⭕               |
+| Wan2.1 I2V 720P              | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers`            | 720p                |    ✅    |        ✅         |    ✅     |              ⭕              |
 **Note**: Wan2.2 TI2V 5B has some quality issues when performing I2V generation. We are working on fixing this issue.
 ### Image Generation Models
-| Model Name       | HuggingFace Model ID                    | Resolutions    |
+| Model Name      | HuggingFace Model ID                | Resolutions    |
-|:-----------------|:----------------------------------------|:---------------|
+| :-------------- | :---------------------------------- | :------------- |
-| FLUX.1-dev       | `black-forest-labs/FLUX.1-dev`          | Any resolution |
+| FLUX.1-dev      | `black-forest-labs/FLUX.1-dev`      | Any resolution |
-| FLUX.2-dev       | `black-forest-labs/FLUX.2-dev`          | Any resolution |
+| FLUX.2-dev      | `black-forest-labs/FLUX.2-dev`      | Any resolution |
-| FLUX.2-Klein     | `black-forest-labs/FLUX.2-klein-4B`     | Any resolution |
+| FLUX.2-Klein    | `black-forest-labs/FLUX.2-klein-4B` | Any resolution |
-| Z-Image-Turbo    | `Tongyi-MAI/Z-Image-Turbo`              | Any resolution |
+| Z-Image-Turbo   | `Tongyi-MAI/Z-Image-Turbo`          | Any resolution |
-| GLM-Image        | `zai-org/GLM-Image`                     | Any resolution |
+| GLM-Image       | `zai-org/GLM-Image`                 | Any resolution |
-| Qwen Image       | `Qwen/Qwen-Image`                       | Any resolution |
+| Qwen Image      | `Qwen/Qwen-Image`                   | Any resolution |
-| Qwen Image 2512  | `Qwen/Qwen-Image-2512`                  | Any resolution |
+| Qwen Image 2512 | `Qwen/Qwen-Image-2512`              | Any resolution |
-| Qwen Image Edit  | `Qwen/Qwen-Image-Edit`                  | Any resolution |
+| Qwen Image Edit | `Qwen/Qwen-Image-Edit`              | Any resolution |
 ## Verified LoRA Examples
@@ -132,21 +134,20 @@ This section lists example LoRAs that have been explicitly tested and verified w
 ### Verified LoRAs by Base Model
-| Base Model       | Supported LoRAs |
+| Base Model      | Supported LoRAs                                                                                                                                    |
-|:-----------------|:----------------|
+| :-------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Wan2.2           | `lightx2v/Wan2.2-Distill-Loras`<br>`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1` |
+| Wan2.2          | `lightx2v/Wan2.2-Distill-Loras`<br>`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1`                                                                          |
-| Wan2.1           | `lightx2v/Wan2.1-Distill-Loras` |
+| Wan2.1          | `lightx2v/Wan2.1-Distill-Loras`                                                                                                                    |
-| Z-Image-Turbo    | `tarn59/pixel_art_style_lora_z_image_turbo`<br>`wcde/Z-Image-Turbo-DeJPEG-Lora` |
+| Z-Image-Turbo   | `tarn59/pixel_art_style_lora_z_image_turbo`<br>`wcde/Z-Image-Turbo-DeJPEG-Lora`                                                                    |
-| Qwen-Image       | `lightx2v/Qwen-Image-Lightning`<br>`flymy-ai/qwen-image-realism-lora`<br>`prithivMLmods/Qwen-Image-HeadshotX`<br>`starsfriday/Qwen-Image-EVA-LoRA` |
+| Qwen-Image      | `lightx2v/Qwen-Image-Lightning`<br>`flymy-ai/qwen-image-realism-lora`<br>`prithivMLmods/Qwen-Image-HeadshotX`<br>`starsfriday/Qwen-Image-EVA-LoRA` |
-| Qwen-Image-Edit  | `ostris/qwen_image_edit_inpainting`<br>`lightx2v/Qwen-Image-Edit-2511-Lightning` |
+| Qwen-Image-Edit | `ostris/qwen_image_edit_inpainting`<br>`lightx2v/Qwen-Image-Edit-2511-Lightning`                                                                   |
-| Flux             | `dvyio/flux-lora-simple-illustration`<br>`XLabs-AI/flux-furry-lora`<br>`XLabs-AI/flux-RealismLora` |
+| Flux            | `dvyio/flux-lora-simple-illustration`<br>`XLabs-AI/flux-furry-lora`<br>`XLabs-AI/flux-RealismLora`                                                 |
 #### Special Requirements
 > [!NOTE]
 > Sliding Tile Attention: Currently, only Hopper GPUs (H100s) are supported.
 ---
 # SGLang diffusion CLI Inference
@@ -158,7 +159,6 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
 - A working SGLang diffusion installation and the `sglang` CLI available in `$PATH`.
 - Python 3.11+ if you plan to use the OpenAI Python SDK.
 ## Supported Arguments
 ### Server Arguments
@@ -173,7 +173,6 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
 - `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP
 - `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP
 ### Sampling Parameters
 - `--prompt {PROMPT}`: Text description for the video you want to generate
@@ -181,7 +180,6 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
 - `--negative-prompt {PROMPT}`: Negative prompt to guide generation away from certain concepts
 - `--seed {SEED}`: Random seed for reproducible generation
 #### Image/Video Configuration
 - `--height {HEIGHT}`: Height of the generated output
@@ -189,7 +187,6 @@ The SGLang-diffusion CLI provides a quick way to access the inference pipeline f
 - `--num-frames {NUM_FRAMES}`: Number of frames to generate
 - `--fps {FPS}`: Frames per second for the saved output, if this is a video-generation task
 #### Output Options
 - `--output-path {PATH}`: Directory to save the generated video
@@ -210,34 +207,31 @@ Example configuration file (config.json):
 ```json
 {
-    "model_path": "FastVideo/FastHunyuan-diffusers",
+  "model_path": "FastVideo/FastHunyuan-diffusers",
-    "prompt": "A beautiful woman in a red dress walking down a street",
+  "prompt": "A beautiful woman in a red dress walking down a street",
-    "output_path": "outputs/",
+  "output_path": "outputs/",
-    "num_gpus": 2,
+  "num_gpus": 2,
-    "sp_size": 2,
+  "sp_size": 2,
-    "tp_size": 1,
+  "tp_size": 1,
-    "num_frames": 45,
+  "num_frames": 45,
-    "height": 720,
+  "height": 720,
-    "width": 1280,
+  "width": 1280,
-    "num_inference_steps": 6,
+  "num_inference_steps": 6,
-    "seed": 1024,
+  "seed": 1024,
-    "fps": 24,
+  "fps": 24,
-    "precision": "bf16",
+  "precision": "bf16",
-    "vae_precision": "fp16",
+  "vae_precision": "fp16",
-    "vae_tiling": true,
+  "vae_tiling": true,
-    "vae_sp": true,
+  "vae_sp": true,
-    "vae_config": {
+  "vae_config": {
-        "load_encoder": false,
+    "load_encoder": false,
-        "load_decoder": true,
+    "load_decoder": true,
-        "tile_sample_min_height": 256,
+    "tile_sample_min_height": 256,
-        "tile_sample_min_width": 256
+    "tile_sample_min_width": 256
-    },
+  },
-    "text_encoder_precisions": [
+  "text_encoder_precisions": ["fp16", "fp16"],
-        "fp16",
+  "mask_strategy_file_path": null,
-        "fp16"
+  "enable_torch_compile": false
    ],
    "mask_strategy_file_path": null,
    "enable_torch_compile": false
 }
 ```
@@ -272,7 +266,6 @@ mask_strategy_file_path: null
 enable_torch_compile: false
 ```
 To see all the options, you can use the `--help` flag:
 ```bash
@@ -305,7 +298,6 @@ sglang serve "${SERVER_ARGS[@]}"
 For detailed API usage, including Image, Video Generation and LoRA management, please refer to the [OpenAI API Documentation](#sglang-diffusion-openai-api).
 ## Generate
 Run a one-off generation task without launching a persistent server.
@@ -346,15 +338,15 @@ SGLang diffusion supports a **diffusers backend** that allows you to run any dif
 ### Arguments
-| Argument | Values | Description |
+| Argument                        | Values                                                | Description                                                                                                                                                   |
-|----------|--------|-------------|
+| ------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `--backend` | `auto` (default), `sglang`, `diffusers` | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline. |
+| `--backend`                     | `auto` (default), `sglang`, `diffusers`               | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline.            |
 | `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines. See [diffusers attention backends](https://huggingface.co/docs/diffusers/main/en/optimization/attention_backends). |
-| `--trust-remote-code` | flag | Required for models with custom pipeline classes (e.g., Ovis). |
+| `--trust-remote-code`           | flag                                                  | Required for models with custom pipeline classes (e.g., Ovis).                                                                                                |
-| `--vae-tiling` | flag | Enable VAE tiling for large image support (decodes tile-by-tile). |
+| `--vae-tiling`                  | flag                                                  | Enable VAE tiling for large image support (decodes tile-by-tile).                                                                                             |
-| `--vae-slicing` | flag | Enable VAE slicing for lower memory usage (decodes slice-by-slice). |
+| `--vae-slicing`                 | flag                                                  | Enable VAE slicing for lower memory usage (decodes slice-by-slice).                                                                                           |
-| `--dit-precision` | `fp16`, `bf16`, `fp32` | Precision for the diffusion transformer. |
+| `--dit-precision`               | `fp16`, `bf16`, `fp32`                                | Precision for the diffusion transformer.                                                                                                                      |
-| `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision for the VAE. |
+| `--vae-precision`               | `fp16`, `bf16`, `fp32`                                | Precision for the VAE.                                                                                                                                        |
 ### Example: Running Ovis-Image-7B
@@ -381,12 +373,12 @@ For pipeline-specific parameters not exposed via CLI, use `diffusers_kwargs` in
 ```json
 {
-    "model_path": "AIDC-AI/Ovis-Image-7B",
+  "model_path": "AIDC-AI/Ovis-Image-7B",
-    "backend": "diffusers",
+  "backend": "diffusers",
-    "prompt": "A beautiful landscape",
+  "prompt": "A beautiful landscape",
-    "diffusers_kwargs": {
+  "diffusers_kwargs": {
-        "cross_attention_kwargs": {"scale": 0.5}
+    "cross_attention_kwargs": { "scale": 0.5 }
-    }
+  }
 }
 ```
@@ -633,7 +625,8 @@ curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
 The server supports dynamic loading, merging, and unmerging of LoRA adapters.
 **Important Notes:**
- Mutual Exclusion: Only one LoRA can be *merged* (active) at a time
+
 - Mutual Exclusion: Only one LoRA can be _merged_ (active) at a time
 - Switching: To switch LoRAs, you must first `unmerge` the current one, then `set` the new one
 - Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost
@@ -644,6 +637,7 @@ Loads one or more LoRA adapters and merges their weights into the model. Support
 **Endpoint:** `POST /v1/set_lora`
 **Parameters:**
 - `lora_nickname` (string or list of strings, required): A unique identifier for the LoRA adapter(s). Can be a single string or a list of strings for multiple LoRAs
 - `lora_path` (string or list of strings/None, optional): Path to the `.safetensors` file(s) or Hugging Face repo ID(s). Required for the first load; optional if re-activating a cached nickname. If a list, must match the length of `lora_nickname`
 - `target` (string or list of strings, optional): Which transformer(s) to apply the LoRA to. If a list, must match the length of `lora_nickname`. Valid values:
@@ -694,21 +688,22 @@ curl -X POST http://localhost:30010/v1/set_lora \
 > [!NOTE]
 > When using multiple LoRAs:
 >
 > - All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length
 > - If `target` or `strength` is a single value, it will be applied to all LoRAs
 > - Multiple LoRAs applied to the same target will be merged in order
 #### Merge LoRA Weights
 Manually merges the currently set LoRA weights into the base model.
 > [!NOTE]
-> `set_lora` automatically performs a merge, so this is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.*
+> `set_lora` automatically performs a merge, so this is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.\*
 **Endpoint:** `POST /v1/merge_lora_weights`
 **Parameters:**
 - `target` (string, optional): Which transformer(s) to merge. One of "all" (default), "transformer", "transformer_2", "critic"
 - `strength` (float, optional): LoRA strength for merge, default 1.0. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
@@ -720,7 +715,6 @@ curl -X POST http://localhost:30010/v1/merge_lora_weights \
  -d '{"strength": 0.8}'
 ```
 #### Unmerge LoRA Weights
 Unmerges the currently active LoRA weights from the base model, restoring it to its original state. This **must** be called before setting a different LoRA.
@@ -768,6 +762,7 @@ curl -sS -X GET "http://localhost:30010/v1/list_loras"
 ```
 Notes:
 - If LoRA is not enabled for the current pipeline, the server will return an error.
 - `num_lora_layers_with_weights` counts only layers that have LoRA weights applied for the active adapter.
@@ -808,16 +803,16 @@ Backend selection is performed by the shared attention layers (e.g. `LocalAttent
 The CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
-| CLI value | Enum value | Notes |
+| CLI value            | Enum value          | Notes                                                                                                                                                                                                                             |
-|---|---|---|
+| -------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `fa` / `fa3` / `fa4` | `FA` | FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.__post_init__`). |
+| `fa` / `fa3` / `fa4` | `FA`                | FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.__post_init__`).                                                                                                                            |
-| `torch_sdpa` | `TORCH_SDPA` | PyTorch `scaled_dot_product_attention`. |
+| `torch_sdpa`         | `TORCH_SDPA`        | PyTorch `scaled_dot_product_attention`.                                                                                                                                                                                           |
-| `sliding_tile_attn` | `SLIDING_TILE_ATTN` | Sliding Tile Attention (STA). Requires `st_attn` and a mask-strategy config file set via the `SGLANG_DIFFUSION_ATTENTION_CONFIG` environment variable. |
+| `sliding_tile_attn`  | `SLIDING_TILE_ATTN` | Sliding Tile Attention (STA). Requires `st_attn` and a mask-strategy config file set via the `SGLANG_DIFFUSION_ATTENTION_CONFIG` environment variable.                                                                            |
-| `sage_attn` | `SAGE_ATTN` | Requires `sageattention`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120 (compute capability 8.0/8.6/8.9/9.0/12.0); see upstream `setup.py`: https://github.com/thu-ml/SageAttention/blob/main/setup.py. |
+| `sage_attn`          | `SAGE_ATTN`         | Requires `sageattention`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120 (compute capability 8.0/8.6/8.9/9.0/12.0); see upstream `setup.py`: https://github.com/thu-ml/SageAttention/blob/main/setup.py. |
-| `sage_attn_3` | `SAGE_ATTN_3` | Requires SageAttention3 installed per upstream instructions. |
+| `sage_attn_3`        | `SAGE_ATTN_3`       | Requires SageAttention3 installed per upstream instructions.                                                                                                                                                                      |
-| `video_sparse_attn` | `VIDEO_SPARSE_ATTN` | Requires `vsa`. |
+| `video_sparse_attn`  | `VIDEO_SPARSE_ATTN` | Requires `vsa`.                                                                                                                                                                                                                   |
-| `vmoba_attn` | `VMOBA_ATTN` | Requires `kernel.attn.vmoba_attn.vmoba`. |
+| `vmoba_attn`         | `VMOBA_ATTN`        | Requires `kernel.attn.vmoba_attn.vmoba`.                                                                                                                                                                                          |
-| `aiter` | `AITER` | Requires `aiter`. |
+| `aiter`              | `AITER`             | Requires `aiter`.                                                                                                                                                                                                                 |
 ## Selection priority
@@ -829,16 +824,16 @@ The selection order in `runtime/layers/attention/selector.py` is:
 ## Platform support matrix
-| Backend | CUDA | ROCm | MPS | Notes |
+| Backend             | CUDA | ROCm | MPS | Notes                                                                                                                                           |
-|---|---:|---:|---:|---|
+| ------------------- | ---: | ---: | --: | ----------------------------------------------------------------------------------------------------------------------------------------------- |
-| `fa` | ✅ | ✅ | ❌ | CUDA requires SM80+ and fp16/bf16. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. |
+| `fa`                |   ✅ |   ✅ |  ❌ | CUDA requires SM80+ and fp16/bf16. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. |
-| `torch_sdpa` | ✅ | ✅ | ✅ | Most compatible option across platforms. |
+| `torch_sdpa`        |   ✅ |   ✅ |  ✅ | Most compatible option across platforms.                                                                                                        |
-| `sliding_tile_attn` | ✅ | ❌ | ❌ | CUDA-only. Requires `st_attn` and `SGLANG_DIFFUSION_ATTENTION_CONFIG`. |
+| `sliding_tile_attn` |   ✅ |   ❌ |  ❌ | CUDA-only. Requires `st_attn` and `SGLANG_DIFFUSION_ATTENTION_CONFIG`.                                                                          |
-| `sage_attn` | ✅ | ❌ | ❌ | CUDA-only (optional dependency). |
+| `sage_attn`         |   ✅ |   ❌ |  ❌ | CUDA-only (optional dependency).                                                                                                                |
-| `sage_attn_3` | ✅ | ❌ | ❌ | CUDA-only (optional dependency). |
+| `sage_attn_3`       |   ✅ |   ❌ |  ❌ | CUDA-only (optional dependency).                                                                                                                |
-| `video_sparse_attn` | ✅ | ❌ | ❌ | CUDA-only. Requires `vsa`. |
+| `video_sparse_attn` |   ✅ |   ❌ |  ❌ | CUDA-only. Requires `vsa`.                                                                                                                      |
-| `vmoba_attn` | ✅ | ❌ | ❌ | CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. |
+| `vmoba_attn`        |   ✅ |   ❌ |  ❌ | CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`.                                                                                             |
-| `aiter` | ✅ | ❌ | ❌ | Requires `aiter`. |
+| `aiter`             |   ✅ |   ❌ |  ❌ | Requires `aiter`.                                                                                                                               |
 ## Usage
@@ -906,7 +901,7 @@ sglang generate --model-path Qwen/Qwen-Image \
 DBCache controls block-level caching behavior:
 | Parameter | Env Variable              | Default | Description                              |
-|-----------|---------------------------|---------|------------------------------------------|
+| --------- | ------------------------- | ------- | ---------------------------------------- |
 | Fn        | `SGLANG_CACHE_DIT_FN`     | 1       | Number of first blocks to always compute |
 | Bn        | `SGLANG_CACHE_DIT_BN`     | 0       | Number of last blocks to always compute  |
 | W         | `SGLANG_CACHE_DIT_WARMUP` | 4       | Warmup steps before caching starts       |
@@ -918,7 +913,7 @@ DBCache controls block-level caching behavior:
 TaylorSeer improves caching accuracy using Taylor expansion:
 | Parameter | Env Variable                  | Default | Description                     |
-|-----------|-------------------------------|---------|---------------------------------|
+| --------- | ----------------------------- | ------- | ------------------------------- |
 | Enable    | `SGLANG_CACHE_DIT_TAYLORSEER` | false   | Enable TaylorSeer calibrator    |
 | Order     | `SGLANG_CACHE_DIT_TS_ORDER`   | 1       | Taylor expansion order (1 or 2) |
@@ -950,7 +945,7 @@ which to use cached results.
 SCM is configured with presets:
 | Preset   | Compute Ratio | Speed    | Quality    |
-|----------|---------------|----------|------------|
+| -------- | ------------- | -------- | ---------- |
 | `none`   | 100%          | Baseline | Best       |
 | `slow`   | ~75%          | ~1.3x    | High       |
 | `medium` | ~50%          | ~2x      | Good       |
@@ -981,7 +976,7 @@ sglang generate --model-path Qwen/Qwen-Image \
 #### SCM Policy
 | Policy    | Env Variable                          | Description                                 |
-|-----------|---------------------------------------|---------------------------------------------|
+| --------- | ------------------------------------- | ------------------------------------------- |
 | `dynamic` | `SGLANG_CACHE_DIT_SCM_POLICY=dynamic` | Adaptive caching based on content (default) |
 | `static`  | `SGLANG_CACHE_DIT_SCM_POLICY=static`  | Fixed caching pattern                       |
@@ -990,7 +985,7 @@ sglang generate --model-path Qwen/Qwen-Image \
 All Cache-DiT parameters can be set via the following environment variables:
 | Environment Variable                | Default | Description                              |
-|-------------------------------------|---------|------------------------------------------|
+| ----------------------------------- | ------- | ---------------------------------------- |
 | `SGLANG_CACHE_DIT_ENABLED`          | false   | Enable Cache-DiT acceleration            |
 | `SGLANG_CACHE_DIT_FN`               | 1       | First N blocks to always compute         |
 | `SGLANG_CACHE_DIT_BN`               | 0       | Last N blocks to always compute          |
@@ -1008,14 +1003,14 @@ All Cache-DiT parameters can be set via the following environment variables:
 SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
-| Model Family | Example Models                            |
+| Model Family | Example Models                       |
-|--------------|-------------------------------------------|
+| ------------ | ------------------------------------ |
-| Wan          | Wan2.1, Wan2.2                            |
+| Wan          | Wan2.1, Wan2.2                       |
-| Flux         | FLUX.1-dev, FLUX.2-dev, FLUX.2-Klein      |
+| Flux         | FLUX.1-dev, FLUX.2-dev, FLUX.2-Klein |
-| Z-Image      | Z-Image-Turbo                             |
+| Z-Image      | Z-Image-Turbo                        |
-| Qwen         | Qwen-Image, Qwen-Image-Edit               |
+| Qwen         | Qwen-Image, Qwen-Image-Edit          |
-| GLM          | GLM-Image                                 |
+| GLM          | GLM-Image                            |
-| Hunyuan      | HunyuanVideo                              |
+| Hunyuan      | HunyuanVideo                         |
 ## Performance Tips
@@ -1075,6 +1070,7 @@ sglang generate \
 ```
 **Parameters:**
 - `--profile`: Enable profiling for the denoising stage
 - `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5)
  - Smaller values reduce trace file size
@@ -1094,6 +1090,7 @@ sglang generate \
 ```
 **Parameters:**
 - `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising
 ### Output Location
@@ -1109,15 +1106,16 @@ The exact output file path will be shown in the console output, for example:
 ### View Traces
 Load and visualize trace files at:
 - https://ui.perfetto.dev/ (recommended)
 - chrome://tracing (Chrome only)
 For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`.
 ### `--perf-dump-path` (Stage/Step Timing Dump)
 Besides profiler traces, you can also dump a lightweight JSON report that contains:
 - stage-level timing breakdown for the full pipeline
 - step-level timing breakdown for the denoising stage (per diffusion step)
@@ -1177,6 +1175,7 @@ nsys profile \
 ```
 **Parameters:**
 - `--delay N`: Wait N seconds before starting capture (skip initialization overhead)
 - `--duration N`: Capture for N seconds (focus on specific stages)
 - `--force-overwrite`: Overwrite existing output files
@@ -1202,16 +1201,19 @@ This guide outlines the requirements for contributing to the SGLang Diffusion mo
 We follow a structured commit message format to maintain a clean history.
 **Format:**
 ```text
 [diffusion] <scope>: <subject>
 ```
 **Examples:**
 - `[diffusion] cli: add --perf-dump-path argument`
 - `[diffusion] scheduler: fix deadlock in batch processing`
 - `[diffusion] model: support Stable Diffusion 3.5`
 **Rules:**
 - **Prefix**: Always start with `[diffusion]`.
 - **Scope** (Optional): `cli`, `scheduler`, `model`, `pipeline`, `docs`, etc.
 - **Subject**: Imperative mood, short and clear (e.g., "add feature" not "added feature").
@@ -1223,11 +1225,13 @@ For PRs that impact **latency**, **throughput**, or **memory usage**, you **shou
 ### How to Generate a Report
 1.  **Baseline**: run the benchmark (for a single generation task)
    ```bash
    $ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path baseline.json
    ```
 2.  **New**: run the same benchmark, without modifying any server_args or sampling_params
    ```bash
    $ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path new.json
    ```
@@ -1238,7 +1242,7 @@ For PRs that impact **latency**, **throughput**, or **memory usage**, you **shou
    ### Performance Comparison Report
    ...
    ```
-4. **Paste**: paste the table into the PR description
+4.  **Paste**: paste the table into the PR description
 ## 3. CI-Based Change Protection
@@ -1260,6 +1264,7 @@ SGLang diffusion uses a modular pipeline architecture built around two key conce
 - **`PipelineStage`**: Modular components (prompt encoding, denoising loop, VAE decoding, etc.)
 To add a new model, you'll need to define:
 1. **`PipelineConfig`**: Static model configurations (paths, precision settings)
 2. **`SamplingParams`**: Runtime generation parameters (prompt, guidance_scale, steps)
 3. **`ComposedPipeline`**: Chain together pipeline stages
@@ -0,0 +1,9 @@
 Image Generation
 ================
 Models for generating images and videos using diffusion.
 .. toctree::
   :maxdepth: 1
   diffusion_models.md
@@ -0,0 +1,14 @@
 Supported Models
 ================
 SGLang supports a wide variety of model architectures for different use cases.
 Browse by category below to find models suited for your needs.
 .. toctree::
   :maxdepth: 2
   text_generation/index
   image_generation/index
   retrieval_ranking/index
   specialized/index
   extending/index
@@ -0,0 +1,11 @@
 Retrieval & Ranking
 ===================
 Models for embeddings, reranking, and classification.
 .. toctree::
   :maxdepth: 1
   embedding_models.md
   rerank_models.md
   classify_models.md
@@ -0,0 +1,9 @@
 Specialized Models
 ==================
 Models for specialized tasks like reward modeling.
 .. toctree::
   :maxdepth: 1
   reward_models.md
@@ -1,28 +1,28 @@
-# Reward Models
+# Reward Models
-
+
-These models output a scalar reward score or classification result, often used in reinforcement learning or content moderation tasks.
+These models output a scalar reward score or classification result, often used in reinforcement learning or content moderation tasks.
-
+
-```{important}
+```{important}
-They are executed with `--is-embedding` and some may require `--trust-remote-code`.
+They are executed with `--is-embedding` and some may require `--trust-remote-code`.
-```
+```
-
+
-## Example launch Command
+## Example launch Command
-
+
-```shell
+```shell
-python3 -m sglang.launch_server \
+python3 -m sglang.launch_server \
-  --model-path Qwen/Qwen2.5-Math-RM-72B \  # example HF/local path
+  --model-path Qwen/Qwen2.5-Math-RM-72B \  # example HF/local path
-  --is-embedding \
+  --is-embedding \
-  --host 0.0.0.0 \
+  --host 0.0.0.0 \
-  --tp-size=4 \                          # set for tensor parallelism
+  --tp-size=4 \                          # set for tensor parallelism
-  --port 30000 \
+  --port 30000 \
-```
+```
-
+
-## Supported models
+## Supported models
-
+
-| Model Family (Reward)                                                     | Example HuggingFace Identifier                              | Description                                                                     |
+| Model Family (Reward)                                                     | Example HuggingFace Identifier                              | Description                                                                     |
-|---------------------------------------------------------------------------|-----------------------------------------------------|---------------------------------------------------------------------------------|
+|---------------------------------------------------------------------------|-----------------------------------------------------|---------------------------------------------------------------------------------|
-| **Llama (3.1 Reward / `LlamaForSequenceClassification`)**                   | `Skywork/Skywork-Reward-Llama-3.1-8B-v0.2`            | Reward model (preference classifier) based on Llama 3.1 (8B) for scoring and ranking responses for RLHF.  |
+| **Llama (3.1 Reward / `LlamaForSequenceClassification`)**                   | `Skywork/Skywork-Reward-Llama-3.1-8B-v0.2`            | Reward model (preference classifier) based on Llama 3.1 (8B) for scoring and ranking responses for RLHF.  |
-| **Gemma 2 (27B Reward / `Gemma2ForSequenceClassification`)**                | `Skywork/Skywork-Reward-Gemma-2-27B-v0.2`             | Derived from Gemma‑2 (27B), this model provides human preference scoring for RLHF and multilingual tasks.  |
+| **Gemma 2 (27B Reward / `Gemma2ForSequenceClassification`)**                | `Skywork/Skywork-Reward-Gemma-2-27B-v0.2`             | Derived from Gemma‑2 (27B), this model provides human preference scoring for RLHF and multilingual tasks.  |
-| **InternLM 2 (Reward / `InternLM2ForRewardMode`)**                         | `internlm/internlm2-7b-reward`                       | InternLM 2 (7B)–based reward model used in alignment pipelines to guide outputs toward preferred behavior.  |
+| **InternLM 2 (Reward / `InternLM2ForRewardMode`)**                         | `internlm/internlm2-7b-reward`                       | InternLM 2 (7B)–based reward model used in alignment pipelines to guide outputs toward preferred behavior.  |
-| **Qwen2.5 (Reward - Math / `Qwen2ForRewardModel`)**                         | `Qwen/Qwen2.5-Math-RM-72B`                           | A 72B math-specialized RLHF reward model from the Qwen2.5 series, tuned for evaluating and refining responses.  |
+| **Qwen2.5 (Reward - Math / `Qwen2ForRewardModel`)**                         | `Qwen/Qwen2.5-Math-RM-72B`                           | A 72B math-specialized RLHF reward model from the Qwen2.5 series, tuned for evaluating and refining responses.  |
-| **Qwen2.5 (Reward - Sequence / `Qwen2ForSequenceClassification`)**          | `jason9693/Qwen2.5-1.5B-apeach`                      | A smaller Qwen2.5 variant used for sequence classification, offering an alternative RLHF scoring mechanism.  |
+| **Qwen2.5 (Reward - Sequence / `Qwen2ForSequenceClassification`)**          | `jason9693/Qwen2.5-1.5B-apeach`                      | A smaller Qwen2.5 variant used for sequence classification, offering an alternative RLHF scoring mechanism.  |
@@ -1,5 +1,7 @@
 # Diffusion Language Models
 > This page covers **text generation** using diffusion-based LLMs. For **image and video generation**, see [Diffusion Models](../image_generation/diffusion_models.md).
 Diffusion language models have shown promise for non-autoregressive text generation with parallel decoding capabilities. Unlike auto-regressive language models, different diffusion language models require different decoding strategies.
 ## Example Launch Command
@@ -25,6 +27,7 @@ threshold: 0.95
 # Default: 32, for LLaDA2MoeModelLM
 block_size: 32
 ```
 ## Example Client Code Snippet
 Just like other supported models, diffusion language models can be used via the REST API or Python client.
@@ -78,6 +81,6 @@ curl -X POST "http://127.0.0.1:30000/generate" \
 Below the supported models are summarized in a table.
-| Model Family                               | Example Model                          | Description                                                                 |
+| Model Family               | Example Model                | Description                                                                                          |
-| ------------------------------------------ | -------------------------------------- | --------------------------------------------------------------------------- |
+| -------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------- |
 | **LLaDA2.0 (mini, flash)** | `inclusionAI/LLaDA2.0-flash` | LLaDA2.0-flash is a diffusion language model featuring a 100B Mixture-of-Experts (MoE) architecture. |
@@ -1,68 +1,70 @@
-# Large Language Models
+# Large Language Models
-
+
-These models accept text input and produce text output (e.g., chat completions). They are primarily large language models (LLMs), some with mixture-of-experts (MoE) architectures for scaling.
+These models accept text input and produce text output (e.g., chat completions). They are primarily large language models (LLMs), some with mixture-of-experts (MoE) architectures for scaling.
-
+
-## Example launch Command
+## Example launch Command
-
+
-```shell
+```shell
-python3 -m sglang.launch_server \
+python3 -m sglang.launch_server \
-  --model-path meta-llama/Llama-3.2-1B-Instruct \  # example HF/local path
+  --model-path meta-llama/Llama-3.2-1B-Instruct \  # example HF/local path
-  --host 0.0.0.0 \
+  --host 0.0.0.0 \
-  --port 30000 \
+  --port 30000 \
-```
+```
-
+
-## Supported models
+## Supported models
-
+
-Below the supported models are summarized in a table.
+Below the supported models are summarized in a table.
-
+
-If you are unsure if a specific architecture is implemented, you can search for it via GitHub. For example, to search for `Qwen3ForCausalLM`, use the expression:
+If you are unsure if a specific architecture is implemented, you can search for it via GitHub. For example, to search for `Qwen3ForCausalLM`, use the expression:
-
+
-```
+```
-repo:sgl-project/sglang path:/^python\/sglang\/srt\/models\// Qwen3ForCausalLM
+repo:sgl-project/sglang path:/^python\/sglang\/srt\/models\// Qwen3ForCausalLM
-```
+```
-
+
-in the GitHub search bar.
+in the GitHub search bar.
-
+
-| Model Family (Variants)             | Example HuggingFace Identifier                     | Description                                                                            |
+| Model Family (Variants)             | Example HuggingFace Identifier                     | Description                                                                            |
-|-------------------------------------|--------------------------------------------------|----------------------------------------------------------------------------------------|
+|-------------------------------------|--------------------------------------------------|----------------------------------------------------------------------------------------|
-| **DeepSeek** (v1, v2, v3/R1)        | `deepseek-ai/DeepSeek-R1`                        | Series of advanced reasoning-optimized models (including a 671B MoE) trained with reinforcement learning; top performance on complex reasoning, math, and code tasks. [SGLang provides Deepseek v3/R1 model-specific optimizations](../basic_usage/deepseek.md) and [Reasoning Parser](../advanced_features/separate_reasoning.ipynb)|
+| **DeepSeek** (v1, v2, v3/R1)        | `deepseek-ai/DeepSeek-R1`                        | Series of advanced reasoning-optimized models (including a 671B MoE) trained with reinforcement learning; top performance on complex reasoning, math, and code tasks. [SGLang provides Deepseek v3/R1 model-specific optimizations](../basic_usage/deepseek.md) and [Reasoning Parser](../advanced_features/separate_reasoning.ipynb)|
-| **Kimi K2** (Thinking, Instruct)    | `moonshotai/Kimi-K2-Instruct`                    | Moonshot AI's 1 trillion parameter MoE model (32B active) with 128K–256K context; state-of-the-art agentic intelligence with stable long-horizon agency across 200–300 sequential tool calls. Features MLA attention and native INT4 quantization. [See Reasoning Parser docs](../advanced_features/separate_reasoning.ipynb)|
+| **Kimi K2** (Thinking, Instruct)    | `moonshotai/Kimi-K2-Instruct`                    | Moonshot AI's 1 trillion parameter MoE model (32B active) with 128K–256K context; state-of-the-art agentic intelligence with stable long-horizon agency across 200–300 sequential tool calls. Features MLA attention and native INT4 quantization. [See Reasoning Parser docs](../advanced_features/separate_reasoning.ipynb)|
-| **Kimi Linear** (48B-A3B)           | `moonshotai/Kimi-Linear-48B-A3B-Instruct`        | Moonshot AI's hybrid linear attention model (48B total, 3B active) with 1M token context; features Kimi Delta Attention (KDA) for up to 6× faster decoding and 75% KV cache reduction vs full attention. |
+| **Kimi Linear** (48B-A3B)           | `moonshotai/Kimi-Linear-48B-A3B-Instruct`        | Moonshot AI's hybrid linear attention model (48B total, 3B active) with 1M token context; features Kimi Delta Attention (KDA) for up to 6× faster decoding and 75% KV cache reduction vs full attention. |
-| **GPT-OSS**       | `openai/gpt-oss-20b`, `openai/gpt-oss-120b`       | OpenAI’s latest GPT-OSS series for complex reasoning, agentic tasks, and versatile developer use cases.|
+| **GPT-OSS**       | `openai/gpt-oss-20b`, `openai/gpt-oss-120b`       | OpenAI’s latest GPT-OSS series for complex reasoning, agentic tasks, and versatile developer use cases.|
-| **Qwen** (3, 3MoE, 3Next, 2.5, 2 series)       | `Qwen/Qwen3-0.6B`, `Qwen/Qwen3-30B-A3B` `Qwen/Qwen3-Next-80B-A3B-Instruct `      | Alibaba’s latest Qwen3 series for complex reasoning, language understanding, and generation tasks; Support for MoE variants along with previous generation 2.5, 2, etc. [SGLang provides Qwen3 specific reasoning parser](../advanced_features/separate_reasoning.ipynb)|
+| **Qwen** (3, 3MoE, 3Next, 2.5, 2 series)       | `Qwen/Qwen3-0.6B`, `Qwen/Qwen3-30B-A3B` `Qwen/Qwen3-Next-80B-A3B-Instruct `      | Alibaba’s latest Qwen3 series for complex reasoning, language understanding, and generation tasks; Support for MoE variants along with previous generation 2.5, 2, etc. [SGLang provides Qwen3 specific reasoning parser](../advanced_features/separate_reasoning.ipynb)|
-| **Llama** (2, 3.x, 4 series)        | `meta-llama/Llama-4-Scout-17B-16E-Instruct`       | Meta's open LLM series, spanning 7B to 400B parameters (Llama 2, 3, and new Llama 4) with well-recognized performance. [SGLang provides Llama-4 model-specific optimizations](../basic_usage/llama4.md)  |
+| **Llama** (2, 3.x, 4 series)        | `meta-llama/Llama-4-Scout-17B-16E-Instruct`       | Meta's open LLM series, spanning 7B to 400B parameters (Llama 2, 3, and new Llama 4) with well-recognized performance. [SGLang provides Llama-4 model-specific optimizations](../basic_usage/llama4.md)  |
-| **Mistral** (Mixtral, NeMo, Small3) | `mistralai/Mistral-7B-Instruct-v0.2`             | Open 7B LLM by Mistral AI with strong performance; extended into MoE (“Mixtral”) and NeMo Megatron variants for larger scale. |
+| **Mistral** (Mixtral, NeMo, Small3) | `mistralai/Mistral-7B-Instruct-v0.2`             | Open 7B LLM by Mistral AI with strong performance; extended into MoE (“Mixtral”) and NeMo Megatron variants for larger scale. |
-| **Gemma** (v1, v2, v3)              | `google/gemma-3-1b-it`                            | Google’s family of efficient multilingual models (1B–27B); Gemma 3 offers a 128K context window, and its larger (4B+) variants support vision input. |
+| **Gemma** (v1, v2, v3)              | `google/gemma-3-1b-it`                            | Google’s family of efficient multilingual models (1B–27B); Gemma 3 offers a 128K context window, and its larger (4B+) variants support vision input. |
-| **Phi** (Phi-1.5, Phi-2, Phi-3, Phi-4, Phi-MoE series) | `microsoft/Phi-4-multimodal-instruct`, `microsoft/Phi-3.5-MoE-instruct` | Microsoft’s Phi family of small models (1.3B–5.6B); Phi-4-multimodal (5.6B) processes text, images, and speech, Phi-4-mini is a high-accuracy text model and Phi-3.5-MoE is a mixture-of-experts model. |
+| **Phi** (Phi-1.5, Phi-2, Phi-3, Phi-4, Phi-MoE series) | `microsoft/Phi-4-multimodal-instruct`, `microsoft/Phi-3.5-MoE-instruct` | Microsoft’s Phi family of small models (1.3B–5.6B); Phi-4-multimodal (5.6B) processes text, images, and speech, Phi-4-mini is a high-accuracy text model and Phi-3.5-MoE is a mixture-of-experts model. |
-| **MiniCPM** (v3, 4B)               | `openbmb/MiniCPM3-4B`                            | OpenBMB’s series of compact LLMs for edge devices; MiniCPM 3 (4B) achieves GPT-3.5-level results in text tasks. |
+| **MiniCPM** (v3, 4B)               | `openbmb/MiniCPM3-4B`                            | OpenBMB’s series of compact LLMs for edge devices; MiniCPM 3 (4B) achieves GPT-3.5-level results in text tasks. |
-| **OLMo** (2, 3) | `allenai/OLMo-3-1125-32B`, `allenai/OLMo-3-32B-Think`, `allenai/OLMo-2-1124-7B-Instruct` | Allen AI’s series of Open Language Models designed to enable the science of language models. |
+| **OLMo** (2, 3) | `allenai/OLMo-3-1125-32B`, `allenai/OLMo-3-32B-Think`, `allenai/OLMo-2-1124-7B-Instruct` | Allen AI’s series of Open Language Models designed to enable the science of language models. |
-| **OLMoE** (Open MoE)               | `allenai/OLMoE-1B-7B-0924`                       | Allen AI’s open Mixture-of-Experts model (7B total, 1B active parameters) delivering state-of-the-art results with sparse expert activation. |
+| **OLMoE** (Open MoE)               | `allenai/OLMoE-1B-7B-0924`                       | Allen AI’s open Mixture-of-Experts model (7B total, 1B active parameters) delivering state-of-the-art results with sparse expert activation. |
-| **MiniMax-M2** (M2, M2.1)                     | `minimax/MiniMax-M2`, `minimax/MiniMax-M2.1`           | MiniMax’s SOTA LLM for coding & agentic workflows. |
+| **MiniMax-M2** (M2, M2.1)                     | `minimax/MiniMax-M2`, `minimax/MiniMax-M2.1`           | MiniMax’s SOTA LLM for coding & agentic workflows. |
-| **StableLM** (3B, 7B)               | `stabilityai/stablelm-tuned-alpha-7b`            | StabilityAI’s early open-source LLM (3B & 7B) for general text generation; a demonstration model with basic instruction-following ability. |
+| **StableLM** (3B, 7B)               | `stabilityai/stablelm-tuned-alpha-7b`            | StabilityAI’s early open-source LLM (3B & 7B) for general text generation; a demonstration model with basic instruction-following ability. |
-| **Command-(R,A)** (Cohere)              | `CohereLabs/c4ai-command-r-v01`, `CohereLabs/c4ai-command-r7b-12-2024`, `CohereLabs/c4ai-command-a-03-2025`                 | Cohere’s open conversational LLM (Command series) optimized for long context, retrieval-augmented generation, and tool use. |
+| **Command-(R,A)** (Cohere)              | `CohereLabs/c4ai-command-r-v01`, `CohereLabs/c4ai-command-r7b-12-2024`, `CohereLabs/c4ai-command-a-03-2025`                 | Cohere’s open conversational LLM (Command series) optimized for long context, retrieval-augmented generation, and tool use. |
-| **DBRX** (Databricks)              | `databricks/dbrx-instruct`                       | Databricks’ 132B-parameter MoE model (36B active) trained on 12T tokens; competes with GPT-3.5 quality as a fully open foundation model. |
+| **DBRX** (Databricks)              | `databricks/dbrx-instruct`                       | Databricks’ 132B-parameter MoE model (36B active) trained on 12T tokens; competes with GPT-3.5 quality as a fully open foundation model. |
-| **Grok** (xAI)                     | `xai-org/grok-1`                                | xAI’s grok-1 model known for vast size(314B parameters) and high quality; integrated in SGLang for high-performance inference. |
+| **Grok** (xAI)                     | `xai-org/grok-1`                                | xAI’s grok-1 model known for vast size(314B parameters) and high quality; integrated in SGLang for high-performance inference. |
-| **ChatGLM** (GLM-130B family)       | `THUDM/chatglm2-6b`                              | Zhipu AI’s bilingual chat model (6B) excelling at Chinese-English dialogue; fine-tuned for conversational quality and alignment. |
+| **ChatGLM** (GLM-130B family)       | `THUDM/chatglm2-6b`                              | Zhipu AI’s bilingual chat model (6B) excelling at Chinese-English dialogue; fine-tuned for conversational quality and alignment. |
-| **InternLM 2** (7B, 20B)           | `internlm/internlm2-7b`                          | Next-gen InternLM (7B and 20B) from SenseTime, offering strong reasoning and ultra-long context support (up to 200K tokens). |
+| **InternLM 2** (7B, 20B)           | `internlm/internlm2-7b`                          | Next-gen InternLM (7B and 20B) from SenseTime, offering strong reasoning and ultra-long context support (up to 200K tokens). |
-| **ExaONE 3** (Korean-English)      | `LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct`           | LG AI Research’s Korean-English model (7.8B) trained on 8T tokens; provides high-quality bilingual understanding and generation. |
+| **ExaONE 3** (Korean-English)      | `LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct`           | LG AI Research’s Korean-English model (7.8B) trained on 8T tokens; provides high-quality bilingual understanding and generation. |
-| **Baichuan 2** (7B, 13B)           | `baichuan-inc/Baichuan2-13B-Chat`                | BaichuanAI’s second-generation Chinese-English LLM (7B/13B) with improved performance and an open commercial license. |
+| **Baichuan 2** (7B, 13B)           | `baichuan-inc/Baichuan2-13B-Chat`                | BaichuanAI’s second-generation Chinese-English LLM (7B/13B) with improved performance and an open commercial license. |
-| **XVERSE** (MoE)                   | `xverse/XVERSE-MoE-A36B`                         | Yuanxiang’s open MoE LLM (XVERSE-MoE-A36B: 255B total, 36B active) supporting ~40 languages; delivers 100B+ dense-level performance via expert routing. |
+| **XVERSE** (MoE)                   | `xverse/XVERSE-MoE-A36B`                         | Yuanxiang’s open MoE LLM (XVERSE-MoE-A36B: 255B total, 36B active) supporting ~40 languages; delivers 100B+ dense-level performance via expert routing. |
-| **SmolLM** (135M–1.7B)            | `HuggingFaceTB/SmolLM-1.7B`                      | Hugging Face’s ultra-small LLM series (135M–1.7B params) offering surprisingly strong results, enabling advanced AI on mobile/edge devices. |
+| **SmolLM** (135M–1.7B)            | `HuggingFaceTB/SmolLM-1.7B`                      | Hugging Face’s ultra-small LLM series (135M–1.7B params) offering surprisingly strong results, enabling advanced AI on mobile/edge devices. |
-| **GLM-4** (Multilingual 9B)        | `ZhipuAI/glm-4-9b-chat`                          | Zhipu’s GLM-4 series (up to 9B parameters) – open multilingual models with support for 1M-token context and even a 5.6B multimodal variant (Phi-4V). |
+| **GLM-4** (Multilingual 9B)        | `ZhipuAI/glm-4-9b-chat`                          | Zhipu’s GLM-4 series (up to 9B parameters) – open multilingual models with support for 1M-token context and even a 5.6B multimodal variant (Phi-4V). |
-| **MiMo** (7B series)               | `XiaomiMiMo/MiMo-7B-RL`                         | Xiaomi's reasoning-optimized model series, leverages Multiple-Token Prediction for faster inference. |
+| **MiMo** (7B series)               | `XiaomiMiMo/MiMo-7B-RL`                         | Xiaomi's reasoning-optimized model series, leverages Multiple-Token Prediction for faster inference. |
-| **ERNIE-4.5** (4.5, 4.5MoE series) | `baidu/ERNIE-4.5-21B-A3B-PT`                    | Baidu's ERNIE-4.5 series which consists of MoE with 47B and 3B active parameters, with the largest model having 424B total parameters, as well as a 0.3B dense model. |
+| **ERNIE-4.5** (4.5, 4.5MoE series) | `baidu/ERNIE-4.5-21B-A3B-PT`                    | Baidu's ERNIE-4.5 series which consists of MoE with 47B and 3B active parameters, with the largest model having 424B total parameters, as well as a 0.3B dense model. |
-| **Arcee AFM-4.5B**               | `arcee-ai/AFM-4.5B-Base`                         | Arcee's foundational model series for real world reliability and edge deployments. |
+| **Arcee AFM-4.5B**               | `arcee-ai/AFM-4.5B-Base`                         | Arcee's foundational model series for real world reliability and edge deployments. |
-| **Persimmon** (8B)               | `adept/persimmon-8b-chat`                         | Adept’s open 8B model with a 16K context window and fast inference; trained for broad usability and licensed under Apache 2.0. |
+| **Persimmon** (8B)               | `adept/persimmon-8b-chat`                         | Adept’s open 8B model with a 16K context window and fast inference; trained for broad usability and licensed under Apache 2.0. |
-| **Solar** (10.7B)               | `upstage/SOLAR-10.7B-Instruct-v1.0`                         | Upstage's 10.7B parameter model, optimized for instruction-following tasks. This architecture incorporates a depth-up scaling methodology, enhancing model performance. |
+| **Solar** (10.7B)               | `upstage/SOLAR-10.7B-Instruct-v1.0`                         | Upstage's 10.7B parameter model, optimized for instruction-following tasks. This architecture incorporates a depth-up scaling methodology, enhancing model performance. |
-| **Tele FLM** (52B-1T)               | `CofeAI/Tele-FLM`                         | BAAI & TeleAI's multilingual model, available in 52-billion and 1-trillion parameter variants. It is a decoder-only transformer trained on ~2T tokens |
+| **Tele FLM** (52B-1T)               | `CofeAI/Tele-FLM`                         | BAAI & TeleAI's multilingual model, available in 52-billion and 1-trillion parameter variants. It is a decoder-only transformer trained on ~2T tokens |
-| **Ling** (16.8B–290B) | `inclusionAI/Ling-lite`, `inclusionAI/Ling-plus` | InclusionAI’s open MoE models. Ling-Lite has 16.8B total / 2.75B active parameters, and Ling-Plus has 290B total / 28.8B active parameters. They are designed for high performance on NLP and complex reasoning tasks. |
+| **Ling** (16.8B–290B) | `inclusionAI/Ling-lite`, `inclusionAI/Ling-plus` | InclusionAI’s open MoE models. Ling-Lite has 16.8B total / 2.75B active parameters, and Ling-Plus has 290B total / 28.8B active parameters. They are designed for high performance on NLP and complex reasoning tasks. |
-| **Granite 3.0, 3.1** (IBM)               | `ibm-granite/granite-3.1-8b-instruct`                          | IBM's open dense foundation models optimized for reasoning, code, and business AI use cases. Integrated with Red Hat and watsonx systems. |
+| **Granite 3.0, 3.1** (IBM)               | `ibm-granite/granite-3.1-8b-instruct`                          | IBM's open dense foundation models optimized for reasoning, code, and business AI use cases. Integrated with Red Hat and watsonx systems. |
-| **Granite 3.0 MoE** (IBM)               | `ibm-granite/granite-3.0-3b-a800m-instruct`                          | IBM’s Mixture-of-Experts models offering strong performance with cost-efficiency. MoE expert routing designed for enterprise deployment at scale. |
+| **Granite 3.0 MoE** (IBM)               | `ibm-granite/granite-3.0-3b-a800m-instruct`                          | IBM’s Mixture-of-Experts models offering strong performance with cost-efficiency. MoE expert routing designed for enterprise deployment at scale. |
-| **GPT-J** (6B)                    | `EleutherAI/gpt-j-6b`                             | EleutherAI's GPT-2-like causal language model (6B) trained on the [Pile](https://pile.eleuther.ai/) dataset. |
+| **GPT-J** (6B)                    | `EleutherAI/gpt-j-6b`                             | EleutherAI's GPT-2-like causal language model (6B) trained on the [Pile](https://pile.eleuther.ai/) dataset. |
-| **Orion** (14B)               | `OrionStarAI/Orion-14B-Base`                         | A series of open-source multilingual large language models by OrionStarAI, pretrained on a 2.5T token multilingual corpus including Chinese, English, Japanese, Korean, etc, and it exhibits superior performance in these languages. |
+| **Orion** (14B)               | `OrionStarAI/Orion-14B-Base`                         | A series of open-source multilingual large language models by OrionStarAI, pretrained on a 2.5T token multilingual corpus including Chinese, English, Japanese, Korean, etc, and it exhibits superior performance in these languages. |
-| **Llama Nemotron Super** (v1, v1.5, NVIDIA) | `nvidia/Llama-3_3-Nemotron-Super-49B-v1`, `nvidia/Llama-3_3-Nemotron-Super-49B-v1_5` | The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents. |
+| **Llama Nemotron Super** (v1, v1.5, NVIDIA) | `nvidia/Llama-3_3-Nemotron-Super-49B-v1`, `nvidia/Llama-3_3-Nemotron-Super-49B-v1_5` | The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents. |
-| **Llama Nemotron Ultra** (v1, NVIDIA) | `nvidia/Llama-3_1-Nemotron-Ultra-253B-v1` | The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents. |
+| **Llama Nemotron Ultra** (v1, NVIDIA) | `nvidia/Llama-3_1-Nemotron-Ultra-253B-v1` | The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents. |
-| **NVIDIA Nemotron Nano 2.0** | `nvidia/NVIDIA-Nemotron-Nano-9B-v2` | The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents. `Nemotron-Nano-9B-v2` is a hybrid Mamba-Transformer language model designed to increase throughput for reasoning workloads while achieving state-of-the-art accuracy compared to similarly-sized models. |
+| **NVIDIA Nemotron Nano 2.0** | `nvidia/NVIDIA-Nemotron-Nano-9B-v2` | The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents. `Nemotron-Nano-9B-v2` is a hybrid Mamba-Transformer language model designed to increase throughput for reasoning workloads while achieving state-of-the-art accuracy compared to similarly-sized models. |
-| **StarCoder2** (3B-15B)               | `bigcode/starcoder2-7b`                         | StarCoder2 is a family of open large language models (LLMs) specialized for code generation and understanding. It is the successor to StarCoder, jointly developed by the BigCode project (a collaboration between Hugging Face, ServiceNow Research, and other contributors). |
+| **StarCoder2** (3B-15B) | `bigcode/starcoder2-7b` | StarCoder2 is a family of open large language models (LLMs) specialized for code generation and understanding. It is the successor to StarCoder, jointly developed by the BigCode project (a collaboration between Hugging Face, ServiceNow Research, and other contributors). |
-| **Jet-Nemotron** | `jet-ai/Jet-Nemotron-2B` | Jet-Nemotron is a new family of hybrid-architecture language models that surpass state-of-the-art open-source full-attention language models, while achieving significant efficiency gains. |
+| **Jet-Nemotron** | `jet-ai/Jet-Nemotron-2B` | Jet-Nemotron is a new family of hybrid-architecture language models that surpass state-of-the-art open-source full-attention language models, while achieving significant efficiency gains. |
-| **Trinity** (Nano, Mini) | `arcee-ai/Trinity-Mini` | Arcee's foundational MoE Trinity family of models, open weights under Apache 2.0. |
+| **Trinity** (Nano, Mini) | `arcee-ai/Trinity-Mini` | Arcee's foundational MoE Trinity family of models, open weights under Apache 2.0. |
 | **Falcon-H1** (0.5B–34B) | `tiiuae/Falcon-H1-34B-Instruct` | TII's hybrid Mamba-Transformer architecture combining attention and state-space models for efficient long-context inference. |
 | **Hunyuan-Large** (389B, MoE) | `tencent/Tencent-Hunyuan-Large` | Tencent's open-source MoE model with 389B total / 52B active parameters, featuring Cross-Layer Attention (CLA) for improved efficiency. |
@@ -0,0 +1,11 @@
 Text Generation
 ===============
 Models for generating text from text or multimodal inputs.
 .. toctree::
   :maxdepth: 1
   generative_models.md
   multimodal_language_models.md
   diffusion_language_models.md
@@ -50,6 +50,8 @@ in the GitHub search bar.
 | **NVIDIA Nemotron Nano 2.0 VL** | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16` | NVIDIA Nemotron Nano v2 VL enables multi-image reasoning and video understanding, along with strong document intelligence, visual Q&A and summarization capabilities. It builds on Nemotron Nano V2, a hybrid Mamba-Transformer LLM, in order to achieve higher inference throughput in long document and video scenarios. | Use `--trust-remote-code`. You may need to adjust `--max-mamba-cache-size` [default is 512] to fit memory constraints. |
 | **Ernie4.5-VL** | `baidu/ERNIE-4.5-VL-28B-A3B-PT`              | Baidu's vision-language models(28B,424B). Support image and video comprehension, and also support thinking.                                                                     |  |
 | **JetVLM** |  | JetVLM is an vision-language model designed for high-performance multimodal understanding and generation tasks built upon Jet-Nemotron. | Coming soon |
 | **Step3-VL** (10B) | `stepfun-ai/Step3-VL-10B` | StepFun's lightweight open-source 10B parameter VLM for multimodal intelligence, excelling in visual perception, complex reasoning, and human alignment. |  |
 | **Qwen3-Omni** | `Qwen/Qwen3-Omni-30B-A3B-Instruct` |  Alibaba's omni-modal MoE model. Currently supports the **Thinker** component (multimodal understanding for text, images, audio, and video), while the **Talker** component (audio generation) is not yet supported. |  |
 ## Video Input Support