[Reasoning + Structured Output] make reasoning compatible with structured output (#12551)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com> Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
This commit is contained in:
@@ -360,6 +360,13 @@
|
||||
"## Native API and SGLang Runtime (SRT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> Note: For native API, as a work-around, you need to set `reasoning` argument to `True` to ensure the model will think before generating the structured output. It's not required for chat-completion API."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
@@ -407,6 +414,7 @@
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": text,\n",
|
||||
" \"reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
@@ -454,6 +462,7 @@
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": text,\n",
|
||||
" \"reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
@@ -482,6 +491,7 @@
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": \"Give me the information of the capital of France.\",\n",
|
||||
" \"reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"temperature\": 0,\n",
|
||||
@@ -519,6 +529,7 @@
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": \"Paris is the capital of\",\n",
|
||||
" \"reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
@@ -547,6 +558,7 @@
|
||||
")\n",
|
||||
"payload = {\n",
|
||||
" \"text\": text,\n",
|
||||
" \"reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"structural_tag\": json.dumps(\n",
|
||||
|
||||
Reference in New Issue
Block a user