From 3e873c2110e7ee98aa66fb821828420fdfa2ea47 Mon Sep 17 00:00:00 2001 From: alp Date: Fri, 4 Sep 2026 14:22:05 -0700 Subject: [PATCH] [Docs] Clarify OpenAI chat template defaults (#32172) Co-authored-by: Xinyuan Tong --- .../advanced_features/server_arguments.mdx | 6 ++++ .../basic_usage/openai_api_completions.mdx | 31 ++++++++++++++++--- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/docs/docs/advanced_features/server_arguments.mdx b/docs/docs/advanced_features/server_arguments.mdx index f6a2bdcc6..897e1dc83 100644 --- a/docs/docs/advanced_features/server_arguments.mdx +++ b/docs/docs/advanced_features/server_arguments.mdx @@ -1121,6 +1121,12 @@ Please consult the documentation below and [server_args.py](https://github.com/s When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool_use', 'rag'), specify which named template to use. If not set, the first available template is used. `None` Type: str + + + `--default-chat-template-kwargs` + A JSON object used as defaults for chat_template_kwargs by Chat Completions, Responses, Anthropic-compatible requests, and chat tokenization. Direct Ollama and embedding paths do not consume these defaults. For ordinary template keys, explicit per-request values take precedence. Keys are specific to the selected model's chat template; reasoning_effort is normalized separately. + `None` + Type: JSON / Dict `--completion-template` diff --git a/docs/docs/basic_usage/openai_api_completions.mdx b/docs/docs/basic_usage/openai_api_completions.mdx index 449b07700..be4e68c3d 100644 --- a/docs/docs/basic_usage/openai_api_completions.mdx +++ b/docs/docs/basic_usage/openai_api_completions.mdx @@ -117,10 +117,10 @@ Some models support internal reasoning or thinking processes that can be exposed #### Basic Usage -To enable reasoning output, you need to: -1. Launch the server with the appropriate reasoning parser -2. Set the model-specific parameter in `chat_template_kwargs` -3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`) +To use reasoning output: +1. Launch the server with the appropriate reasoning parser. +2. Use the model-specific parameter in `chat_template_kwargs` when you need to override the model's chat-template default. +3. Optionally use `separate_reasoning: False` to keep reasoning in the response content instead of returning it separately (defaults to `True`). **Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content. @@ -176,9 +176,30 @@ Thus, the total count is **3**. **Answer:** 3. ``` -Setting `"enable_thinking": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter. +Setting `"enable_thinking": False` disables reasoning for Qwen3. Omitting the parameter follows the model's chat-template default; Qwen3 enables thinking by default. Qwen3-Thinking models always generate reasoning content and do not support the `enable_thinking` parameter. +#### Set a Default for OpenAI Chat Completions + +If most OpenAI chat-completion requests should use the same chat-template mode, set a default when launching the server: + +```bash Command +python -m sglang.launch_server \ + --model-path Qwen/Qwen3-4B \ + --reasoning-parser qwen3 \ + --default-chat-template-kwargs '{"enable_thinking": false}' +``` + +`--default-chat-template-kwargs` accepts a JSON object and merges it into each OpenAI chat-completion request's `chat_template_kwargs`. For ordinary template keys such as `enable_thinking`, the precedence is: + +1. An explicit value in the request's `chat_template_kwargs` +2. The value in `--default-chat-template-kwargs` +3. The model's chat-template default + +`reasoning_effort` follows a separate normalization path, so the precedence list above does not apply to it. Configure `reasoning_effort` in either the server defaults or individual requests, not both. + +The accepted keys are model-specific. For example, Qwen3 uses `enable_thinking`, while other model families may use `thinking` or may not expose a reasoning toggle. The reasoning parser controls how reasoning tokens are separated from the response; it does not by itself enable reasoning in the chat template. + #### Logit Bias Support SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where: