minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] Sync docs_new with legacy docs and update migration redirects (#23337)

Browse Source 
Co-authored-by: Mingyi <wisclmy0611@gmail.com>
This commit is contained in:
zijiexia
2026-04-21 00:15:17 -07:00
committed by GitHub
co-authored by Mingyi
parent f63def8510
commit 900aad5f72
179 changed files with 16014 additions and 8162 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/advanced_features/quantization.mdx
+437 -25
View File
@@ -20,11 +20,276 @@ or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on
popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization
to guard against abnormal quantization loss regressions.
## Platform Compatibility
The following table summarizes quantization method support across NVIDIA and AMD GPUs, Ascend NPUs.
<table>
<thead>
<tr>
<th>Method</th>
<th>NVIDIA GPUs</th>
<th>AMD GPUs (MI300X/MI325X/MI350X)</th>
<th>Ascend NPUs (A2/A3)</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>fp8</code></td>
<td>Yes</td>
<td>Yes</td>
<td>WIP</td>
<td>Aiter or Triton backend on AMD</td>
</tr>
<tr>
<td><code>mxfp4</code></td>
<td>Yes</td>
<td>Yes</td>
<td>WIP</td>
<td>Requires CDNA3/CDNA4 with MXFP support; uses Aiter</td>
</tr>
<tr>
<td><code>blockwise_int8</code></td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
<td>Triton-based, works on both platforms</td>
</tr>
<tr>
<td><code>w8a8_int8</code></td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
<td></td>
</tr>
<tr>
<td><code>w8a8_fp8</code></td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
<td>Aiter or Triton FP8 on AMD</td>
</tr>
<tr>
<td><code>awq</code></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Uses Triton dequantize on AMD (vs. optimized CUDA kernels on NVIDIA). Uses CANN kernels on Ascend</td>
</tr>
<tr>
<td><code>gptq</code></td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Uses Triton or vLLM kernels on AMD. Uses CANN kernels on Ascend</td>
</tr>
<tr>
<td><code>compressed-tensors</code></td>
<td>Yes</td>
<td>Yes</td>
<td>Partial</td>
<td>Aiter paths for FP8/MoE on AMD. Uses CANN kernels on Ascend, <code>FP8</code> not supported yet</td>
</tr>
<tr>
<td><code>quark</code></td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
<td>AMD Quark quantization; Aiter GEMM paths on AMD</td>
</tr>
<tr>
<td><code>auto-round</code></td>
<td>Yes</td>
<td>Yes</td>
<td>Partial</td>
<td>Platform-agnostic (Intel auto-round). Uses CANN kernels on Ascend</td>
</tr>
<tr>
<td><code>quark_int4fp8_moe</code></td>
<td>No</td>
<td>Yes</td>
<td>No</td>
<td>AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4)</td>
</tr>
<tr>
<td><code>awq_marlin</code></td>
<td>Yes</td>
<td>No</td>
<td>No</td>
<td>Marlin kernels are CUDA-only</td>
</tr>
<tr>
<td><code>gptq_marlin</code></td>
<td>Yes</td>
<td>No</td>
<td>No</td>
<td>Marlin kernels are CUDA-only</td>
</tr>
<tr>
<td><code>gguf</code></td>
<td>Yes</td>
<td>No</td>
<td>WIP</td>
<td>CUDA-only kernels in sgl-kernel</td>
</tr>
<tr>
<td><code>modelopt</code> / <code>modelopt_fp8</code></td>
<td>Yes (Hopper/SM90+)</td>
<td>No</td>
<td>No</td>
<td><a href="https://github.com/NVIDIA/Model-Optimizer">NVIDIA ModelOpt</a>; requires NVIDIA hardware</td>
</tr>
<tr>
<td><code>modelopt_fp4</code></td>
<td>Yes (Blackwell/SM100+)</td>
<td>No</td>
<td>No</td>
<td><a href="https://github.com/NVIDIA/Model-Optimizer">NVIDIA ModelOpt</a>; native FP4 on Blackwell (B200, GB200)</td>
</tr>
<tr>
<td><code>petit_nvfp4</code></td>
<td>No</td>
<td>Yes (MI250/MI300X/MI325X)</td>
<td>No</td>
<td>Enables NVFP4 on ROCm via <a href="https://github.com/causalflow-ai/petit-kernel">Petit</a>; use <code>modelopt_fp4</code> on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See <a href="https://lmsys.org/blog/2025-09-21-petit-amdgpu/">LMSYS blog</a> and <a href="https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html">AMD ROCm blog</a>.</td>
</tr>
<tr>
<td><code>bitsandbytes</code></td>
<td>Yes</td>
<td>Experimental</td>
<td>No</td>
<td>Depends on bitsandbytes ROCm support</td>
</tr>
<tr>
<td><code>torchao</code> (<code>int4wo</code>, etc.)</td>
<td>Yes</td>
<td>Partial</td>
<td>No</td>
<td><code>int4wo</code> not supported on AMD; other methods may work</td>
</tr>
<tr>
<td><code>modelslim</code></td>
<td>No</td>
<td>No</td>
<td>Yes</td>
<td>Ascend quantization; Uses CANN kernels</td>
</tr>
</tbody>
</table>
On AMD, several of these methods use [Aiter](https://github.com/ROCm/aiter) for acceleration -- set `SGLANG_USE_AITER=1` where noted. See [AMD GPU setup](../hardware-platforms/amd_gpu) for installation and configuration details.
On Ascend, various layers quantization configurations are supported, see [Ascend NPU quantization](../hardware-platforms/ascend-npus/ascend_npu_quantization) for details.
## GEMM Backends for FP4/FP8 Quantization
<Note>
Backend selection is supported only for **blockwise FP8** and **NVFP4** GEMM. When running FP8 or FP4 quantized models, you can select the GEMM backend via `--fp8-gemm-backend` and `--fp4-gemm-backend`.
</Note>
### `--fp8-gemm-backend` (Blockwise FP8 GEMM)
<table>
<thead>
<tr>
<th>Backend</th>
<th>Hardware</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>auto</code></td>
<td>All</td>
<td>Auto-selects based on hardware</td>
</tr>
<tr>
<td><code>deep_gemm</code></td>
<td>SM90, SM100</td>
<td>JIT-compiled; enabled when DeepGEMM is installed</td>
</tr>
<tr>
<td><code>flashinfer_trtllm</code></td>
<td>SM100</td>
<td>FlashInfer TensorRT-LLM backend; optimal for low-latency</td>
</tr>
<tr>
<td><code>flashinfer_cutlass</code></td>
<td>SM100/120</td>
<td>FlashInfer CUTLASS groupwise FP8 GEMM</td>
</tr>
<tr>
<td><code>flashinfer_deepgemm</code></td>
<td>SM90</td>
<td>Uses swapAB optimization for small M dimensions in decoding</td>
</tr>
<tr>
<td><code>cutlass</code></td>
<td>SM90, SM100/120</td>
<td>sgl-kernel CUTLASS</td>
</tr>
<tr>
<td><code>triton</code></td>
<td>All</td>
<td>Fallback; widely compatible</td>
</tr>
<tr>
<td><code>aiter</code></td>
<td>ROCm</td>
<td>AMD AITER backend</td>
</tr>
</tbody>
</table>
**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM90/SM100/120); 4) AITER (AMD); 5) Triton. **Exception:** SM120 always resolves to Triton.
### `--fp4-gemm-backend` (NVFP4 GEMM)
<table>
<thead>
<tr>
<th>Backend</th>
<th>Hardware</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>auto</code></td>
<td>SM100/120</td>
<td>Auto-selects: <code>flashinfer_cudnn</code> on SM120; <code>flashinfer_cutlass</code> on SM100</td>
</tr>
<tr>
<td><code>cutlass</code></td>
<td>SM100/120</td>
<td>SGLang CUTLASS kernel</td>
</tr>
<tr>
<td><code>flashinfer_cutlass</code></td>
<td>SM100/120</td>
<td>FlashInfer CUTLASS backend</td>
</tr>
<tr>
<td><code>flashinfer_cudnn</code></td>
<td>SM100/120 (CUDA 13+, cuDNN 9.15+)</td>
<td>FlashInfer cuDNN backend; used on SM120 for performance</td>
</tr>
<tr>
<td><code>flashinfer_trtllm</code></td>
<td>SM100</td>
<td>FlashInfer TensorRT-LLM backend</td>
</tr>
</tbody>
</table>
When FlashInfer is unavailable for NVFP4, the SGLang CUTLASS kernel is used as an automatic fallback.
## Offline Quantization
To load already quantized models, simply load the model weights and config. **Again, if the model has been quantized offline,
there's no need to add `--quantization` argument when starting the engine. The quantization method will be parsed from the
downloaded Hugging Face config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.**
downloaded Hugging Face or msModelSlim config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.**
```bash Command
python3 -m sglang.launch_server \
@@ -194,23 +459,85 @@ python3 -m sglang.launch_server \
#### Using [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)
NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware. SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment.
NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware.
**Offline vs. Online Quantization:**
SGLang supports two modes for ModelOpt.
* **Offline Quantization (pre-quantized):**
* **Usage:** Download a pre-quantized model from Hugging Face or run `hf_ptq.py` once to create a new quantized checkpoint. Then load this quantized checkpoint.
* **Pros:** Fast server startup, quantization can be validated before deployment, efficient resource usage.
* **Cons:** Requires an extra preparation step.
* **Online Quantization (quant and serve):**
* **Usage:** Load a standard BF16/FP16 model and add a flag. The engine applies quantization *on startup*.
* **Pros:** Convenient (no new checkpoint needed).
* **Cons:** **High startup time**, increases VRAM usage during initialization (risk of OOM).
The following sections guide you through using the Offline path: loading pre-quantized models or creating your own checkpoints.
##### Using Pre-Quantized Checkpoints
If a model is already quantized (e.g., from Hugging Face), you can load it directly.
* **FP8 Models:**
Use `--quantization modelopt_fp8`.
```bash Command
python3 -m sglang.launch_server \
--model-path nvidia/Llama-3.1-8B-Instruct-FP8 \
--quantization modelopt_fp8 \
--port 30000
```
* **FP4 Models:**
Use `--quantization modelopt_fp4`.
```bash Command
python3 -m sglang.launch_server \
--model-path nvidia/Llama-3.3-70B-Instruct-NVFP4 \
--quantization modelopt_fp4 \
--port 30000
```
##### Creating Your Own Quantized Checkpoints
If a pre-quantized checkpoint is not available for your model, you can create one using NVIDIA Model Optimizer's `hf_ptq.py` script.
**Why quantize?**
- Reduce VRAM usage
- Higher throughput and lower latency
- More flexible deployment (on smaller GPUs)
**What can be quantized?**
- The entire model
- MLP layers only
- KV cache
**Key options in `hf_ptq.py`:**
`--qformat`: Quantization formats `fp8`, `nvfp4`, `nvfp4_mlp_only`
`--kv_cache_qformat`: KV cache quantization format (default: `fp8`)
**Note:** The default `kv_cache_qformat` may not be optimal for all use cases. Consider setting this explicitly.
**Hardware requirements:** Hopper and higher are recommended. Insufficient GPU memory may cause weight offloading, resulting in extremely long quantization time.
For detailed usage and supported model architectures, see [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq).
SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment.
##### Installation
First, install ModelOpt. You can either install it directly or as an optional SGLang dependency:
First, install ModelOpt:
```bash Command
# Option 1: Install ModelOpt directly
pip install nvidia-modelopt
# Option 2: Install SGLang with ModelOpt support (recommended)
pip install sglang[modelopt]
```
##### Quantization and Export Workflow
SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow:
SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow. Run from the SGLang repository root (see [modelopt_quantize_and_export.py](https://github.com/sgl-project/sglang/blob/main/examples/usage/modelopt_quantize_and_export.py)):
```bash Command
# Quantize and export a model using ModelOpt FP8 quantization
@@ -219,7 +546,7 @@ python examples/usage/modelopt_quantize_and_export.py quantize \
--export-dir ./quantized_tinyllama_fp8 \
--quantization-method modelopt_fp8
# For FP4 quantization
# For FP4 quantization (requires Blackwell GPU)
python examples/usage/modelopt_quantize_and_export.py quantize \
--model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
--export-dir ./quantized_tinyllama_fp4 \
@@ -275,25 +602,39 @@ python -m sglang.launch_server \
--port 30000 --host 0.0.0.0
```
Or using the Python API:
Or using the Python API (use the same path as `modelopt_export_path` from the quantize step):
```python Example
import sglang as sgl
# Deploy exported ModelOpt quantized model
llm = sgl.Engine(
model_path="./quantized_tinyllama_fp8",
quantization="modelopt"
)
def main():
# Deploy exported ModelOpt quantized model
# Path must match modelopt_export_path from quantize step (e.g., ./exported_model)
llm = sgl.Engine(
model_path="./exported_model",
quantization="modelopt",
)
# Run inference
prompts = ["Hello, how are you?", "What is the capital of France?"]
sampling_params = {"temperature": 0.8, "top_p": 0.95, "max_new_tokens": 100}
outputs = llm.generate(prompts, sampling_params)
# Run inference
prompts = [
"Hello, how are you?",
"What is the capital of France?",
]
sampling_params = {
"temperature": 0.8,
"top_p": 0.95,
"max_new_tokens": 100,
}
outputs = llm.generate(prompts, sampling_params)
for i, output in enumerate(outputs):
print(f"Prompt: {prompts[i]}")
print(f"Output: {output['text']}")
if __name__ == "__main__":
main()
for i, output in enumerate(outputs):
print(f"Prompt: {prompts[i]}")
print(f"Output: {output.outputs[0].text}")
```
##### Advanced Features
@@ -311,7 +652,7 @@ python examples/usage/modelopt_quantize_and_export.py quantize \
# The checkpoint can be reused for future quantization runs and skip calibration
```
**Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly:
**Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly. See [LoadConfig](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/configs/load_config.py) for the full API:
```python Example
from sglang.srt.configs.device_config import DeviceConfig
@@ -330,7 +671,7 @@ load_config = LoadConfig(
modelopt_export_path="./exported_model",
)
# Load and export the model
# Load and export the model (DeviceConfig defaults to device="cuda")
model_loader = get_model_loader(load_config, model_config)
model_loader.load_model(model_config=model_config, device_config=DeviceConfig())
```
@@ -343,6 +684,74 @@ model_loader.load_model(model_config=model_config, device_config=DeviceConfig())
- **Calibration-based**: Uses calibration datasets for optimal quantization quality
- **Production Ready**: Enterprise-grade quantization with NVIDIA support
#### Using [ModelSlim](https://gitcode.com/Ascend/msmodelslim)
MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware.
- **Installation**
```bash Command
# Clone repo and install msmodelslim:
git clone https://gitcode.com/Ascend/msmodelslim.git
cd msmodelslim
bash install.sh
```
- **LLM quantization**
Download the original floating-point weights of the large model. Taking Qwen3-32B as an example, you can go to [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the huggingface model card).
> Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).
_Traditional quantification methods require the preparation of calibration data files (```.jsonl``` formats) for calibration in the quantification process._
```bash Command
Qwen3-32B/ # floating-point model downloaded from official HF (or modelscope) repo
msmodelslim/ # msmodelslim repo
|----- lab_calib # calibration date folder (put your dataset here in ```.jsonl``` format or use pre-prepared ones)
|----- some file (such as laos_calib.jsonl)
|----- lab_practice # best practice folder with configs for quantization
|----- model folder (such as qwen3_5_moe folder) # folder with quantization configs
|----- quant_config (such as qwen3_5_moe_w8a8.yaml) # quantization config
|----- another folders
output_folder/ # generated by below command
|----- quant_model_weights-00001-of-0001.safetensors # quantized weights
|----- quant_model_description.json # file with description of the quantization methods for each layer (```W4A4_DYNAMIC```, etc.)
|----- another files (such as config.json, tokenizer.json, etc.)
```
Run quantization using one-click quantization (recommended):
```bash Command
msmodelslim quant \
--model_path ${MODEL_PATH} \
--save_path ${SAVE_PATH} \
--device npu:0,1 \
--model_type Qwen3-32B \
--quant_type w8a8 \
--trust_remote_code True
```
- **Usage Example**
```bash Command
python3 -m sglang.launch_server \
--model-path $PWD/Qwen3-32B-w8a8 \
--port 30000 --host 0.0.0.0
```
- **Available Quantization Methods**:
- [x] ```W4A4_DYNAMIC``` linear with online quantization of activations
- [x] ```W8A8``` linear with offline quantization of activations
- [x] ```W8A8_DYNAMIC``` linear with online quantization of activations
- [x] ```W4A4_DYNAMIC``` MOE with online quantization of activations
- [x] ```W4A8_DYNAMIC``` MOE with online quantization of activations
- [x] ```W8A8_DYNAMIC``` MOE with online quantization of activations
- [ ] ```W4A8``` linear TBD
- [ ] ```W4A16``` linear TBD
- [ ] ```W48A16``` linear TBD
- [ ] ```W4A16``` MoE in progress
- [ ] ```W8A16``` MoE in progress
- [ ] ```KV Cache``` in progress
- [ ] ```Attention``` in progress
For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/README.md) section in ModelSLim repo.
## Online Quantization
To enable online quantization, you can simply specify `--quantization` in the command line. For example, you can launch the server with the following command to enable `FP8` quantization for model `meta-llama/Meta-Llama-3.1-8B-Instruct`:
@@ -381,7 +790,7 @@ python3 -m sglang.launch_server \
### `quark_int4fp8_moe` online quantization method
SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/fused_moe_triton/layer.py) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8.
SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/v0.4.8/python/sglang/srt/layers/moe/fused_moe_triton/layer.py#L271) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8.
Other layers (e.g. projections in the attention layers) have their weights quantized online to float8 directly.
@@ -390,6 +799,9 @@ Other layers (e.g. projections in the attention layers) have their weights quant
- [GPTQModel](https://github.com/ModelCloud/GPTQModel)
- [LLM Compressor](https://github.com/vllm-project/llm-compressor/)
- [NVIDIA Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer)
- [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq)
- [Petit: NVFP4 on ROCm](https://github.com/causalflow-ai/petit-kernel) — [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/), [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html)
- [Torchao: PyTorch Architecture Optimization](https://github.com/pytorch/ao)
- [vLLM Quantization](https://docs.vllm.ai/en/latest/quantization/)
- [auto-round](https://github.com/intel/auto-round)
- [ModelSlim](https://gitcode.com/Ascend/msmodelslim)