minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

add doc for quantized kv cache (#14348)

Browse Source 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
Co-authored-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
This commit is contained in:
b8zhong
2025-12-04 13:01:05 -08:00
committed by GitHub
co-authored by Brayden Zhong Ho-Ren Chuang
parent 922756aaa1
commit 88d1bab537
3 changed files with 164 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/quantization.md
+1 -1
View File
@@ -12,7 +12,7 @@ on-the-fly to convert high-precision weights into a lower-precision format.
**Note: For better performance, usability and convenience, offline quantization is recommended over online quantization.**
If you use a pre-quantized model, do not add `--quantization` to enable online quantization at the same time.
For popular pre-quantized models, please visit [Unsloth](https://huggingface.co/unsloth), [ModelCloud](https://huggingface.co/collections/ModelCloud/vortex-673743382af0a52b2a8b9fe2)
For popular pre-quantized models, please visit [Unsloth](https://huggingface.co/unsloth), [NVIDIA ModelOpt](https://huggingface.co/collections/nvidia/inference-optimized-checkpoints-with-model-optimizer)
or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on HF for some
popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization
to guard against abnormal quantization loss regressions.