Add new Mintlify documentation site (docs_new/) (#23001)
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
co-authored by
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
parent
575fdc2c4c
commit
a3291b5654
@@ -0,0 +1,54 @@
|
||||
---
|
||||
title: "Custom Chat Template"
|
||||
metatags:
|
||||
description: "SGLang custom chat templates: JSON and Jinja formats for OpenAI-compatible API server. Override tokenizer defaults."
|
||||
---
|
||||
**NOTE**: There are two chat template systems in SGLang project. This document is about setting a custom chat template for the OpenAI-compatible API server (defined at [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/conversation.py)). It is NOT related to the chat template used in the SGLang language frontend (defined at [chat_template.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/lang/chat_template.py)).
|
||||
|
||||
By default, the server uses the chat template specified in the model tokenizer from Hugging Face.
|
||||
It should just work for most official models such as Llama-2/Llama-3.
|
||||
|
||||
If needed, you can also override the chat template when launching the server:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-2-7b-chat-hf \
|
||||
--port 30000 \
|
||||
--chat-template llama-2
|
||||
```
|
||||
|
||||
If the chat template you are looking for is missing, you are welcome to contribute it or load it from a file.
|
||||
|
||||
## JSON Format
|
||||
|
||||
You can load the JSON format, which is defined by `conversation.py`.
|
||||
|
||||
```json Config
|
||||
{
|
||||
"name": "my_model",
|
||||
"system": "<|im_start|>system",
|
||||
"user": "<|im_start|>user",
|
||||
"assistant": "<|im_start|>assistant",
|
||||
"sep_style": "CHATML",
|
||||
"sep": "<|im_end|>",
|
||||
"stop_str": ["<|im_end|>", "<|im_start|>"]
|
||||
}
|
||||
```
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-2-7b-chat-hf \
|
||||
--port 30000 \
|
||||
--chat-template ./my_model_template.json
|
||||
```
|
||||
|
||||
## Jinja Format
|
||||
|
||||
You can also use the [Jinja template format](https://huggingface.co/docs/transformers/main/en/chat_templating) as defined by Hugging Face Transformers.
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-2-7b-chat-hf \
|
||||
--port 30000 \
|
||||
--chat-template ./my_model_template.jinja
|
||||
```
|
||||
Reference in New Issue
Block a user