Add new Mintlify documentation site (docs_new/) (#23001)
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
co-authored by
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
parent
575fdc2c4c
commit
a3291b5654
@@ -0,0 +1,317 @@
|
||||
---
|
||||
title: "Reasoning Parser"
|
||||
metatags:
|
||||
description: "SGLang reasoning parser: separate thinking content from output for DeepSeek R1, Qwen3, Kimi K2, GPT-OSS reasoning models."
|
||||
---
|
||||
SGLang supports parsing reasoning content out from "normal" content for reasoning models such as [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1).
|
||||
|
||||
## Supported Models & Parsers
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning tags</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parser</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[DeepSeek‑R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseek-r1`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Supports all variants (R1, R1-0528, R1-Distill)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[DeepSeek‑V3 series](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseek-v3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Including [DeepSeek‑V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp). Supports `thinking` parameter</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Standard Qwen3 models](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Supports `enable_thinking` parameter</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen3-Thinking models](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen3` or `qwen3-thinking`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Always generates thinking content</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`◁think▷` … `◁/think▷`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`kimi_k2`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Uses special thinking delimiters. Also requires `--tool-call-parser kimi_k2` for tool use.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[GPT OSS](https://huggingface.co/openai/gpt-oss-120b)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<|channel|>analysis<|message|>` … `<|end|>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`gpt-oss`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
### Model-Specific Behaviors
|
||||
|
||||
**DeepSeek-R1 Family:**
|
||||
- DeepSeek-R1: No `<think>` start tag, jumps directly to thinking content
|
||||
- DeepSeek-R1-0528: Generates both `<think>` start and `</think>` end tags
|
||||
- Both are handled by the same `deepseek-r1` parser
|
||||
|
||||
**DeepSeek-V3 Family:**
|
||||
- DeepSeek-V3.1/V3.2: Hybrid model supporting both thinking and non-thinking modes, use the `deepseek-v3` parser and `thinking` parameter (NOTE: not `enable_thinking`)
|
||||
|
||||
**Qwen3 Family:**
|
||||
- Standard Qwen3 (e.g., Qwen3-2507): Use `qwen3` parser, supports `enable_thinking` in chat templates
|
||||
- Qwen3-Thinking (e.g., Qwen3-235B-A22B-Thinking-2507): Use `qwen3` or `qwen3-thinking` parser, always thinks
|
||||
|
||||
**Kimi K2:**
|
||||
- Kimi K2 Thinking: Uses special `◁think▷` and `◁/think▷` tags. For agentic tool use, also specify `--tool-call-parser kimi_k2`.
|
||||
|
||||
**GPT OSS:**
|
||||
- GPT OSS: Uses special `<|channel|>analysis<|message|>` and `<|end|>` tags
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
### Launching the Server
|
||||
|
||||
|
||||
Specify the `--reasoning-parser` option.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
Note that `--reasoning-parser` defines the parser used to interpret responses.
|
||||
|
||||
|
||||
### OpenAI Compatible API
|
||||
|
||||
Using the OpenAI compatible API, the contract follows the [DeepSeek API design](https://api-docs.deepseek.com/guides/reasoning_model) established with the release of DeepSeek-R1:
|
||||
|
||||
- `reasoning_content`: The content of the CoT.
|
||||
- `content`: The content of the final answer.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# Initialize OpenAI-like client
|
||||
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1")
|
||||
model_name = client.models.list().data[0].id
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "What is 1+3?",
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### Non-Streaming Request
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response_non_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
stream=False, # Non-streaming
|
||||
extra_body={"separate_reasoning": True},
|
||||
)
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(response_non_stream.choices[0].message.reasoning_content)
|
||||
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(response_non_stream.choices[0].message.content)
|
||||
```
|
||||
|
||||
#### Streaming Request
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
stream=True, # Non-streaming
|
||||
extra_body={"separate_reasoning": True},
|
||||
)
|
||||
|
||||
reasoning_content = ""
|
||||
content = ""
|
||||
for chunk in response_stream:
|
||||
if chunk.choices[0].delta.content:
|
||||
content += chunk.choices[0].delta.content
|
||||
if chunk.choices[0].delta.reasoning_content:
|
||||
reasoning_content += chunk.choices[0].delta.reasoning_content
|
||||
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(reasoning_content)
|
||||
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(content)
|
||||
```
|
||||
|
||||
Optionally, you can buffer the reasoning content to the last reasoning chunk (or the first chunk after the reasoning content).
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
stream=True, # Non-streaming
|
||||
extra_body={"separate_reasoning": True, "stream_reasoning": False},
|
||||
)
|
||||
|
||||
reasoning_content = ""
|
||||
content = ""
|
||||
for chunk in response_stream:
|
||||
if chunk.choices[0].delta.content:
|
||||
content += chunk.choices[0].delta.content
|
||||
if chunk.choices[0].delta.reasoning_content:
|
||||
reasoning_content += chunk.choices[0].delta.reasoning_content
|
||||
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(reasoning_content)
|
||||
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(content)
|
||||
```
|
||||
|
||||
The reasoning separation is enable by default when specify .
|
||||
**To disable it, set the `separate_reasoning` option to `False` in request.**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response_non_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
stream=False, # Non-streaming
|
||||
extra_body={"separate_reasoning": False},
|
||||
)
|
||||
|
||||
print_highlight("==== Original Output ====")
|
||||
print_highlight(response_non_stream.choices[0].message.content)
|
||||
```
|
||||
|
||||
### SGLang Native API
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
|
||||
input = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
|
||||
gen_url = f"http://localhost:{port}/generate"
|
||||
gen_data = {
|
||||
"text": input,
|
||||
"sampling_params": {
|
||||
"skip_special_tokens": False,
|
||||
"max_new_tokens": 1024,
|
||||
"temperature": 0.6,
|
||||
"top_p": 0.95,
|
||||
},
|
||||
}
|
||||
gen_response = requests.post(gen_url, json=gen_data).json()["text"]
|
||||
|
||||
print_highlight("==== Original Output ====")
|
||||
print_highlight(gen_response)
|
||||
|
||||
parse_url = f"http://localhost:{port}/separate_reasoning"
|
||||
separate_reasoning_data = {
|
||||
"text": gen_response,
|
||||
"reasoning_parser": "deepseek-r1",
|
||||
}
|
||||
separate_reasoning_response_json = requests.post(
|
||||
parse_url, json=separate_reasoning_data
|
||||
).json()
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(separate_reasoning_response_json["reasoning_text"])
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(separate_reasoning_response_json["text"])
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
### Offline Engine API
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
from sglang.srt.parser.reasoning_parser import ReasoningParser
|
||||
from sglang.utils import print_highlight
|
||||
|
||||
llm = sgl.Engine(model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
|
||||
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
|
||||
input = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
sampling_params = {
|
||||
"max_new_tokens": 1024,
|
||||
"skip_special_tokens": False,
|
||||
"temperature": 0.6,
|
||||
"top_p": 0.95,
|
||||
}
|
||||
result = llm.generate(prompt=input, sampling_params=sampling_params)
|
||||
|
||||
generated_text = result["text"] # Assume there is only one prompt
|
||||
|
||||
print_highlight("==== Original Output ====")
|
||||
print_highlight(generated_text)
|
||||
|
||||
parser = ReasoningParser("deepseek-r1")
|
||||
reasoning_text, text = parser.parse_non_stream(generated_text)
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(reasoning_text)
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(text)
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
llm.shutdown()
|
||||
```
|
||||
|
||||
## Supporting New Reasoning Model Schemas
|
||||
|
||||
For future reasoning models, you can implement the reasoning parser as a subclass of `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py` and specify the reasoning parser for new reasoning model schemas accordingly.
|
||||
Reference in New Issue
Block a user