minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Update quantization.md with new model resources (#13677)

Browse Source
This commit is contained in:
赵晨阳
2025-11-20 15:50:16 -08:00
committed by GitHub
parent 3f1cfd87b6
commit c56fc42430
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/quantization.md
+5 -1
View File
@@ -12,7 +12,7 @@ on-the-fly to convert high-precision weights into a lower-precision format.
**Note: For better performance, usability and convenience, offline quantization is recommended over online quantization.** **Note: For better performance, usability and convenience, offline quantization is recommended over online quantization.**
If you use a pre-quantized model, do not add `--quantization` to enable online quantization at the same time. If you use a pre-quantized model, do not add `--quantization` to enable online quantization at the same time.
For popular pre-quantized models, please visit [ModelCloud](https://huggingface.co/collections/ModelCloud/vortex-673743382af0a52b2a8b9fe2) For popular pre-quantized models, please visit [Unsloth](https://huggingface.co/unsloth), [ModelCloud](https://huggingface.co/collections/ModelCloud/vortex-673743382af0a52b2a8b9fe2)
or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on HF for some or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on HF for some
popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization
to guard against abnormal quantization loss regressions. to guard against abnormal quantization loss regressions.
@@ -40,6 +40,10 @@ python3 -m sglang.launch_server \
### Examples of Offline Model Quantization ### Examples of Offline Model Quantization
#### Using [Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide)
We strongly suggest the use of Unsloth to quantize and load the model. Please refer to [SGLang Deployment & Inference Guide with Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide).
#### Using [auto-round](https://github.com/intel/auto-round) #### Using [auto-round](https://github.com/intel/auto-round)
```bash ```bash