[Feat] sgl-router: Render Kimi-K3 through sglang-processor (#43608)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kan Wu
2026-10-13 23:40:18 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 6f646563a2
commit 53a24c448e
39 changed files with 7448 additions and 249 deletions
@@ -183,8 +183,10 @@ PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
When a new spec needs more, extend `serving_chat()` in the script (one place):
- **More server state:** set the extra attribute that `__init__` resolves, for
example `_dsv41_default_reasoning_effort`.
- **Tokenizes through `apply_chat_template(tokenize=True)`:** the recorder sees no
text. Record the template's text output instead.
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
recorded by the template's text output. Its `render_request` returns the
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
Pin `revision` to a commit hash and regenerate only on purpose.
+15 -13
View File
@@ -410,10 +410,11 @@ text, as does any model whose template fails to load or render.
Some chats additionally forward the rendered tokens to the engine as
`input_ids`, retaining the original messages, so the engine skips
re-tokenizing. How many depends on the model's renderer. DeepSeek-V4's and
V4.1's native encoders and GLM-5.3's template are fixture-verified against SGLang's
request normalization, so they forward every chat except multimodal ones and those with
caller-provided `input_ids`. Renderers without that verification (HF Jinja templates, Kimi-K3)
re-tokenizing. How many depends on the model's renderer. DeepSeek-V4's, V4.1's
and Kimi-K3's native encoders and GLM-5.3's template are fixture-verified against
SGLang's request normalization, so they forward every chat except multimodal ones
and those with caller-provided `input_ids`. Renderers without that verification
(HF Jinja templates)
forward only plain text chat requests (string `content`, no tools, no template
kwargs or reasoning controls or historical `reasoning_content`, no assistant
continuation, no consecutive users or non-leading system turns) and warn
@@ -508,12 +509,13 @@ own route as sent: workers that disagree on those args or never
reported them, `--completion-template`, `return_hidden_states` and other
`sglext` outputs, `echo` or `logprobs` without a router tokenizer, and bodies
SGLang would reject. Chat also needs a renderer verified against SGLang (the
`AllText` scope, DeepSeek-V4, V4.1 and GLM-5.3 today), router `--default-chat-template-kwargs`
`AllText` scope: DeepSeek-V4, V4.1, GLM-5.3 and Kimi-K3 today), router `--default-chat-template-kwargs`
equal to the workers', a reasoning parser the processor reproduces
(`deepseek-v4`, `deepseek-v41`, `glm45`) or none, and no media yet. Tools are served with `tool_choice`
(`deepseek-v4`, `deepseek-v41`, `glm45`, `kimi_k3`) or none, and no media yet. Tools are served with `tool_choice`
`auto` or `none`, non-strict, with standard JSON-schema types, and a tool
parser the processor reproduces (`deepseekv4`, `deepseekv41`); `required`, named and strict
tools take a constraint from the engine's xgrammar, so they go to the engine.
parser the processor reproduces (`deepseekv4`, `deepseekv41`, `kimi_k3`); `required`, named and strict
tools take a constraint from the engine's xgrammar, so they go to the engine
(`kimi_k3`'s `auto` tag is reproduced).
With `--tool-call-parser glm47`, a chat with tools also goes to the engine: its
grammar needs `ebnf_full_assistant`, which `/generate` does not accept.
The router reads the model's `config.json` and `generation_config.json` beside
@@ -596,13 +598,13 @@ tools keep the engine's route (see above).
## Kimi-K3
Kimi-K3 renders through dynamo-render's native XTML formatter with SGLang's
request semantics (reasoning controls, tools, `response_format`, continuations)
and the checkpoint's chunked tiktoken encoding. `--tokenizer-path` accepts a
Kimi-K3 renders through `rust/sglang-processor` (dynamo-render's XTML formatter
with SGLang's request semantics) and is tokenized with the checkpoint's chunked
tiktoken encoding; shapes the formatter renders differently go to the worker.
`--tokenizer-path` accepts a
local `tiktoken.model` or an HF repo id, whose `tiktoken.model`, `config.json`
and `tokenizer_config.json` are downloaded when it has no `tokenizer.json`.
An explicit null `thinking_effort` with thinking enabled is not representable
in the pinned formatter and falls back to engine-side rendering.
With the cookbook's `kimi_k3` parsers, text chats are served through `/generate`.
## HTTP/2
@@ -4,7 +4,7 @@
//! Chat rendering via dynamo-render for cache-aware routing and input ID forwarding.
//!
//! Mirrors SGLang reasoning controls and assistant continuations; DeepSeek-V4,
//! V4.1 and GLM-5.3 render through `sglang_processor`. Forwarding remains guarded
//! V4.1, GLM-5.3 and Kimi-K3 render through `sglang_processor`. Forwarding remains guarded
//! until parity has been verified for each request shape.
use std::collections::HashMap;
@@ -41,9 +41,8 @@ pub struct ChatFormatter {
defaults: ChatTemplateKwargs,
/// Stripped from a separately tokenized continuation prefix, as SGLang does.
bos_token: Option<String>,
/// DeepSeek-V4, V4.1 and GLM-5.3 render through `sglang_processor`.
/// DeepSeek-V4, V4.1, GLM-5.3 and Kimi-K3 render through `sglang_processor`.
processor: Option<ProcessorFormatter>,
is_kimi_k3: bool,
/// Worker defaults, merged before model-specific effort conversion.
worker_defaults: ChatTemplateKwargs,
}
@@ -184,13 +183,12 @@ impl ChatFormatter {
defaults,
bos_token,
processor,
is_kimi_k3: false,
worker_defaults: ChatTemplateKwargs::new(),
}))
}
/// dynamo-render's native Kimi-K3 XTML formatter, wrapped in SGLang's request
/// semantics (`kimi::normalize`) and the checkpoint's chunked tokenization.
/// Kimi-K3's native XTML formatter, rendered by `sglang_processor` and
/// tokenized with the checkpoint's chunked encoding.
pub fn kimi_native(model_type: Option<&str>, model_id: &str) -> Option<Self> {
let model_type = model_type.map(str::to_lowercase);
let PromptFormatter::OAI(formatter) =
@@ -199,8 +197,7 @@ impl ChatFormatter {
formatter,
defaults: HashMap::new(),
bos_token: Some("[BOS]".into()),
processor: None,
is_kimi_k3: true,
processor: Some(ProcessorFormatter::KimiK3),
worker_defaults: ChatTemplateKwargs::new(),
})
}
@@ -246,7 +243,6 @@ impl ChatFormatter {
} else {
is_deepseek_v4.then_some(ProcessorFormatter::DeepSeekV4(DeepSeekV4Profile::Preview))
},
is_kimi_k3: false,
worker_defaults: ChatTemplateKwargs::new(),
})
}
@@ -258,7 +254,8 @@ impl ChatFormatter {
.is_some_and(ProcessorFormatter::always_thinks)
}
/// DeepSeek-V4, V4.1 and GLM-5.3 are fixture-verified against SGLang for every text chat.
/// DeepSeek-V4, V4.1, GLM-5.3 and Kimi-K3 are fixture-verified against SGLang for every
/// text chat.
pub fn forwarding_scope(&self) -> super::ForwardingScope {
match self.processor {
Some(_) => super::ForwardingScope::AllText,
@@ -266,6 +263,12 @@ impl ChatFormatter {
}
}
/// Kimi-K3 encodes in the checkpoint's chunks, and its OpenAI layer applies K3's rules
/// (`chat_encoding_spec`).
pub fn is_kimi_k3(&self) -> bool {
matches!(self.processor, Some(ProcessorFormatter::KimiK3))
}
/// Apply the workers' `--default-chat-template-kwargs`; they fill keys the
/// request leaves unset, and a default `reasoning_effort` acts as the request's.
pub fn with_defaults(mut self, defaults: &ChatTemplateKwargs) -> Self {
@@ -330,19 +333,13 @@ impl ChatFormatter {
}
// serving_chat first pops the request kwarg into request.reasoning_effort,
// then merges worker defaults. Jinja receives those merged kwargs over
// the effective effort; Kimi derives thinking_effort only if still absent.
// the effective effort.
kwargs.remove("reasoning_effort");
for (key, value) in &self.worker_defaults {
kwargs.entry(key.clone()).or_insert_with(|| value.clone());
}
if let Some(effort) = self.effective_effort(request) {
if self.is_kimi_k3 {
if matches!(effort.as_str(), Some("low" | "high" | "max")) {
kwargs.entry("thinking_effort".into()).or_insert(effort);
}
} else {
kwargs.entry("reasoning_effort".into()).or_insert(effort);
}
kwargs.entry("reasoning_effort".into()).or_insert(effort);
}
for (key, value) in &self.defaults {
kwargs.entry(key.clone()).or_insert_with(|| value.clone());
@@ -354,12 +351,11 @@ impl ChatFormatter {
/// separately (`_handle_last_assistant_message`).
fn render_parts(&self, request: &JsonValue) -> Result<(RenderedPrompt, String)> {
if let Some(formatter) = &self.processor {
let (prompt, prefix) = formatter
return formatter
.render_request(request.clone(), &self.worker_defaults)
.context("render chat")?;
return Ok((RenderedPrompt::text(prompt), prefix));
.context("render chat");
}
let mut kwargs = self.template_kwargs(request)?;
let kwargs = self.template_kwargs(request)?;
let continuing = request["continue_final_message"] == true;
let mut messages: Vec<JsonValue> = request["messages"]
.as_array()
@@ -367,16 +363,13 @@ impl ChatFormatter {
.iter()
.map(engine_message)
.collect();
if self.is_kimi_k3 {
super::kimi::normalize(request, &mut messages, &mut kwargs)?;
}
let mut prefix = String::new();
if let Some(last) = messages.last_mut().filter(|m| m["role"] == "assistant") {
if let Some(content) = last["content"].as_str() {
if continuing {
prefix = content.to_owned();
messages.pop();
} else if !self.is_kimi_k3 {
} else {
*last = serde_json::json!({"role": "user", "content": content});
}
}
@@ -387,7 +380,6 @@ impl ChatFormatter {
request,
messages,
kwargs,
is_kimi_k3: self.is_kimi_k3,
})
.context("render chat template")?;
Ok((prompt, prefix))
@@ -396,13 +388,13 @@ impl ChatFormatter {
pub fn encode(&self, tokenizer: &Tokenizer, request: &JsonValue) -> Result<Vec<u32>> {
let (prompt, prefix) = self.render_parts(request)?;
let mut ids = match prompt.encode_segments() {
Some(segments) if self.is_kimi_k3 => super::kimi::encode(tokenizer, &segments)?,
Some(segments) if self.is_kimi_k3() => super::kimi::encode(tokenizer, &segments)?,
Some(segments) => tokenizer.encode_segments(&segments)?.token_ids().to_vec(),
None => super::adapter::encode(tokenizer, prompt.as_str())?,
};
if !prefix.is_empty() {
// SGLang encodes the assistant prefix separately and removes its leading BOS.
let mut suffix = if self.is_kimi_k3 {
let mut suffix = if self.is_kimi_k3() {
super::kimi::encode(tokenizer, &[EncodeSegment::control(&prefix)])?
} else {
super::adapter::encode(tokenizer, &prefix)?
@@ -481,7 +473,6 @@ struct ChatRequest<'a> {
/// Normalized copy of `request["messages"]`.
messages: Vec<JsonValue>,
kwargs: ChatTemplateKwargs,
is_kimi_k3: bool,
}
impl OAIChatLikeRequest for ChatRequest<'_> {
@@ -502,9 +493,6 @@ impl OAIChatLikeRequest for ChatRequest<'_> {
if tools.as_array().is_none_or(|t| t.is_empty()) {
return None;
}
if self.is_kimi_k3 {
return Some(Value::from_serialize(tools));
}
let mut tools = tools.clone();
// SGLang renders only the named tool for a function `tool_choice`.
if let Some(name) = self.request["tool_choice"]["function"]["name"].as_str() {
@@ -515,25 +503,13 @@ impl OAIChatLikeRequest for ChatRequest<'_> {
may_be_fix_tool_schema(tools)
}
fn tool_choice(&self) -> Option<Value> {
if self.is_kimi_k3 {
self.kwargs.get("tool_choice").map(Value::from_serialize)
} else {
self.request.get("tool_choice").map(Value::from_serialize)
}
self.request.get("tool_choice").map(Value::from_serialize)
}
fn reasoning_effort(&self) -> Option<Value> {
self.kwargs
.get("reasoning_effort")
.map(Value::from_serialize)
}
/// Only Kimi-K3 renders `response_format`; elsewhere the engine enforces
/// it by constrained decoding and never renders it.
fn response_format(&self) -> Option<Value> {
self.is_kimi_k3
.then(|| self.kwargs.get("response_format"))
.flatten()
.map(Value::from_serialize)
}
fn should_add_generation_prompt(&self) -> bool {
true
}
+3 -98
View File
@@ -1,106 +1,11 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 The SGLang Authors
// SPDX-License-Identifier: Apache-2.0
//! Kimi-K3 request semantics from `serving_chat.py` and the checkpoint's
//! `tokenization_kimi.py`, applied around dynamo-render's native formatter.
//! Kimi-K3's tokenization: `sglang_processor` renders its prompts, and the
//! checkpoint's `tokenization_kimi.py` encodes them in chunks.
use anyhow::{ensure, Result};
use anyhow::Result;
use dynamo_tokenizers::{EncodeSegment, Tokenizer};
use serde_json::{json, Value};
use super::chat_formatter::ChatTemplateKwargs;
pub(super) fn normalize(
request: &Value,
messages: &mut [Value],
kwargs: &mut ChatTemplateKwargs,
) -> Result<()> {
// Dynamo reads `reasoning_effort` and treats a non-bool `thinking` as true;
// the checkpoint ignores the former and uses Python truthiness for the latter.
kwargs.remove("reasoning_effort");
let thinking = kwargs
.get("thinking")
.is_none_or(|v| minijinja::Value::from_serialize(v).is_true());
kwargs.insert("thinking".into(), thinking.into());
ensure!(
!thinking || !kwargs.get("thinking_effort").is_some_and(Value::is_null),
"Kimi null thinking_effort requires engine-side rendering"
);
for message in messages.iter_mut() {
if message["role"] == "developer" {
message["role"] = "system".into();
}
if let Some(parts) = message["content"].as_array_mut() {
parts.retain(|part| matches!(part["type"].as_str(), Some("text" | "image_url")));
}
for call in message["tool_calls"].as_array_mut().into_iter().flatten() {
let args = &mut call["function"]["arguments"];
if let Some(parsed) = args
.as_str()
.and_then(|s| serde_json::from_str::<Value>(s).ok())
.filter(Value::is_object)
{
*args = parsed;
}
neutralize(args);
}
neutralize(&mut message["content"]);
if let Some(reasoning) = message.get_mut("reasoning_content") {
neutralize(reasoning);
}
}
let has_tools = std::iter::once(request)
.chain(messages.iter().filter(|m| m["role"] == "system"))
.any(|m| m["tools"].as_array().is_some_and(|t| !t.is_empty()));
if has_tools && matches!(request["tool_choice"].as_str(), Some("none" | "required")) {
kwargs
.entry("tool_choice".into())
.or_insert_with(|| request["tool_choice"].clone());
}
// The checkpoint renders only these; a named choice is constrained decoding.
if !matches!(
kwargs.get("tool_choice").and_then(Value::as_str),
Some("none" | "required")
) {
kwargs.remove("tool_choice");
}
if let Some(mut format) = request
.get("response_format")
.filter(|v| !v.is_null())
.cloned()
{
// protocol.py lifts a legacy top-level `schema` into `json_schema`.
if format["type"] == "json_schema" && format["json_schema"].is_null() {
if let Some(mut schema) = format.as_object_mut().and_then(|f| f.remove("schema")) {
if let Some(props) = schema.get_mut("properties").and_then(Value::as_object_mut) {
props.remove("strict");
}
format["json_schema"] = json!({"schema": schema});
}
}
kwargs.entry("response_format".into()).or_insert(format);
}
if let Some(schema) = kwargs.get("response_schema").cloned() {
if let Some(format) = kwargs
.get_mut("response_format")
.filter(|f| f["type"] == "json_schema")
{
format["json_schema"] = json!({"schema": schema});
}
}
Ok(())
}
fn neutralize(value: &mut Value) {
match value {
Value::String(text) => {
*text = text.replace("<|kimi_image_placeholder|>", "<| kimi_image_placeholder |>")
}
Value::Array(values) => values.iter_mut().for_each(neutralize),
Value::Object(values) => values.values_mut().for_each(neutralize),
_ => {}
}
}
/// `tokenization_kimi.py` encodes 400k-char windows, each split after 25k
/// consecutive (non-)whitespace chars, and BPE is not chunk-invariant.
+5 -1
View File
@@ -154,6 +154,7 @@ impl TokenizerRegistry {
.unwrap_or_default(),
// Read from the chat formatter below.
always_thinks: false,
chat_encoding_spec: None,
};
me.inner.insert(m.id.clone(), t);
me.stats = stats;
@@ -168,6 +169,9 @@ impl TokenizerRegistry {
Ok(Some(formatter)) => {
let formatter = formatter.with_defaults(&m.default_chat_template_kwargs);
me.chat_model.always_thinks = formatter.always_thinks();
if formatter.is_kimi_k3() {
me.chat_model.chat_encoding_spec = Some("kimi_k3".into());
}
me.formatters
.insert(m.id.clone(), Arc::new(ChatFormatterEntry::new(formatter)));
tracing::info!(model = %m.id, "dynamo-render chat rendering enabled");
@@ -189,7 +193,7 @@ impl TokenizerRegistry {
and SGLANG_DSV4_REASONING_EFFORT or SGLANG_DSV41_REASONING_EFFORT"),
ForwardingScope::Guarded => tracing::warn!(model = %m.id,
"UNVERIFIED input_ids forwarding: router rendering is verified against SGLang only \
for DeepSeek-V4, V4.1 and GLM-5.3, so this model forwards only guarded request shapes (plain text \
for DeepSeek-V4, V4.1, GLM-5.3 and Kimi-K3, so this model forwards only guarded shapes (plain text \
chat). Requires the workers' model files and --default-chat-template-kwargs; \
worker parser overrides, content-format detection, and conversation-template stop \
strings are not replicated. Pass --disable-input-ids-forwarding unless you have \
@@ -26,12 +26,6 @@ fn kimi_tokens_match_sglang() {
.repeat(repeat as usize)
.into();
}
// Dynamo rejects a system turn with both content and tools, which SGLang
// renders; the router leaves that shape to the worker.
if case["name"] == "developer_tools" {
assert!(formatter.encode(&tokenizer, &request).is_err());
continue;
}
let ids = formatter.encode(&tokenizer, &request).unwrap();
let mut hash = Sha256::new();
for id in &ids {
@@ -47,4 +41,11 @@ fn kimi_tokens_match_sglang() {
}
let request = json!({"messages":[{"role":"user","content":"hi"}], "chat_template_kwargs":{"thinking_effort":null}});
assert!(formatter.encode(&tokenizer, &request).is_err());
// SGLang's `_validate_request` rejects a required tool_choice without tools.
let request = json!({"messages":[{"role":"user","content":"hi"}], "tool_choice":"required"});
assert!(formatter.encode(&tokenizer, &request).is_err());
// Media renders on the engine; the router never forwards ids for it.
let image = json!({"type":"image_url","image_url":{"url":"https://example.com/a.png"}});
let request = json!({"messages":[{"role":"user","content":[image]}]});
assert!(formatter.encode(&tokenizer, &request).is_err());
}
@@ -7,14 +7,12 @@
{"name":"unsupported_effort","request":{"messages":[{"role":"user","content":"hello <|open|> <|kimi_image_placeholder|>"}],"reasoning_effort":"medium"},"token_count":275,"sha256":"43f237fef9c404e15ba9ac9a4fc948b6301b8a0bf5336576dd5e90c46d5c1cb8"},
{"name":"tool_none","request":{"messages":[{"role":"user","content":"hello <|open|> <|kimi_image_placeholder|>"}],"tools":[{"type":"function","function":{"name":"lookup","parameters":{"type":"object"}}}],"tool_choice":"none"},"token_count":523,"sha256":"925e846f2a2686ecbb6ee899b15155305459bd01b31833c8e4c522d6655a5bae"},
{"name":"named_tool","request":{"messages":[{"role":"user","content":"hello <|open|> <|kimi_image_placeholder|>"}],"tools":[{"type":"function","function":{"name":"lookup","parameters":{"type":"object"}}}],"tool_choice":{"type":"function","function":{"name":"lookup"}}},"token_count":426,"sha256":"6d87edaec38985be9cbbba896e68ceb6c6cf6c9fa0b0d3addf0b05045511347c"},
{"name":"choice_without_tools","request":{"messages":[{"role":"user","content":"hello <|open|> <|kimi_image_placeholder|>"}],"tool_choice":"required"},"token_count":275,"sha256":"43f237fef9c404e15ba9ac9a4fc948b6301b8a0bf5336576dd5e90c46d5c1cb8"},
{"name":"developer_tools","request":{"messages":[{"role":"developer","content":"policy","tools":[{"type":"function","function":{"name":"lookup","parameters":{"type":"object"}}}]},{"role":"user","content":"hello <|open|> <|kimi_image_placeholder|>"}],"tool_choice":"required"},"token_count":614,"sha256":"16ee5d4d7cf852ce4a5cb7a079f42fcae517200704a7a99f5e12572976e6c50c"},
{"name":"schema_alias","request":{"messages":[{"role":"user","content":"hello <|open|> <|kimi_image_placeholder|>"}],"response_format":{"type":"json_schema","schema":{"type":"object","properties":{"answer":{"type":"string"},"strict":{"default":true}}}}},"token_count":530,"sha256":"4978c30e0da93740589ec2d193f06ade089301d6af32ec6f2b39a4792cad5e17"},
{"name":"format_override","request":{"messages":[{"role":"user","content":"hello <|open|> <|kimi_image_placeholder|>"}],"response_format":{"type":"json_object"},"chat_template_kwargs":{"response_format":{"type":"json_schema","json_schema":{"name":"answer","schema":{"type":"string"}}}}},"token_count":494,"sha256":"d78ae98b5fd5998c75a7cbcf7ca253a169c61fdea7b6549c0d959c44e9f486d5"},
{"name":"continuation","request":{"messages":[{"role":"user","content":"hello <|open|> <|kimi_image_placeholder|>"},{"role":"assistant","content":"[BOS]partial <|open|> <|kimi_image_placeholder|>"}],"continue_final_message":true},"token_count":309,"sha256":"583362e9c2e28226e52ac6ca3eb4fb2d2c312653d995bcac291ff94154cdd1a7"},
{"name":"assistant_history","request":{"messages":[{"role":"user","content":"hello","name":"ignored"},{"role":"assistant","reasoning_content":"why <|kimi_image_placeholder|>","content":"answer"}]},"token_count":299,"sha256":"0480fa42372ee2cd1bc0653a1d3bebef5eee0b2acc7aac47960b05d14768dc8a"},
{"name":"tool_result","request":{"messages":[{"role":"assistant","tool_calls":[{"id":"c1","type":"function","function":{"name":"lookup","arguments":"{\"x\":\"<|kimi_image_placeholder|>\"}"}}]},{"role":"tool","tool_call_id":"c1","content":"result"}]},"token_count":362,"sha256":"4f85e03e0f4e9e7909040afbac4432a57fc1b01ce1c90b13d0c365cf4155336e"},
{"name":"image","request":{"messages":[{"role":"user","content":[{"type":"text","text":"hi <|kimi_image_placeholder|>"},{"type":"image_url","image_url":{"url":"https://example.com/a.png"}}]}]},"token_count":269,"sha256":"c22f63a7138b758acf9925aea4264e6a3469b0ea50c4b1d16325f825476e966d"},
{"name":"long_run","request":{"messages":[{"role":"user","content":"界hello"}]},"repeat":4168,"token_count":16915,"sha256":"05651835990fb23205bd5731718edd7c0b246d30e4f84ad345671156d0c0be4c"},
{"name":"long_segment","request":{"messages":[{"role":"user","content":"hello "}]},"repeat":66668,"token_count":266910,"sha256":"0d1ec109069b74c82c0af95b2685972dc56f6c381639a62f9993384407354ec3"},
{"name":"null_thinking","request":{"messages":[{"role":"user","content":"hello"}],"chat_template_kwargs":{"thinking":null,"thinking_effort":null,"tool_choice":"specified"}},"token_count":24,"sha256":"e7a5f51a4704edb223f44493ca6f637d008177112d612fb57267a9e647480d3f"},
@@ -21,7 +21,8 @@ use sgl_router::proxy::Proxy;
use sgl_router::server::app::build_router;
use sgl_router::server::app_context::AppContext;
use sgl_router::tokenizer::TokenizerRegistry;
use sgl_router::workers::{Worker, WorkerRegistry};
use sgl_router::workers::{EngineProfile, WireProtocol, Worker, WorkerRegistry};
use sglang_processor::openai::OpenAiSettings;
use std::sync::Arc;
use std::time::Duration;
use tower::ServiceExt;
@@ -73,18 +74,24 @@ fn build_ctx(url: String) -> Arc<AppContext> {
}
fn build_ctx_with_config(url: String, cfg: Config) -> Arc<AppContext> {
build_ctx_with_profile(url, cfg, WireProtocol::default().into())
}
/// [`build_ctx_with_config`] whose worker introspected as `profile`.
fn build_ctx_with_profile(url: String, cfg: Config, profile: EngineProfile) -> Arc<AppContext> {
// The handler tokenizes via the AppContext's registry (which carries the V4
// encoder); the RoundRobin policy itself needs no tokenizer.
let tokenizers = Arc::new(TokenizerRegistry::load_from_config(&cfg).unwrap());
assert!(tokenizers.has_chat_formatter(MODEL));
let registry = Arc::new(WorkerRegistry::default());
let _ = registry.add(WorkerSpec {
let spec = WorkerSpec {
id: WorkerId(url.clone()),
url,
mode: WorkerMode::Plain,
model_ids: vec![ModelId(MODEL.into())],
..Default::default()
});
};
let _ = registry.add_with_cb(spec, None, profile);
let policies = Arc::new(build_registry_with_defaults(&cfg).unwrap());
let proxy = Arc::new(Proxy::new(Duration::from_secs(5)).unwrap());
Arc::new(AppContext::new(cfg, tokenizers, proxy, registry, policies))
@@ -487,3 +494,44 @@ async fn kimi_ids_forward_with_engine_rendering_fallback() {
assert_eq!(without_minted_rid(captured(&mock)), request);
}
}
/// With the workers' OpenAI settings and K3's parsers, a Kimi-K3 chat is
/// lowered to `/generate` on the router's ids, with K3's OpenAI-layer rules.
#[tokio::test]
async fn kimi_chat_lowers_to_generate() {
let mock = MockWorker::start(vec![]).await;
let fixture = kimi_fixture::tokenizer();
let mut cfg = config();
let path = fixture.path().join("tiktoken.model");
cfg.model.tokenizer_path = Some(path.display().to_string());
let settings = OpenAiSettings {
reasoning_parser: Some("kimi_k3".into()),
tool_call_parser: Some("kimi_k3".into()),
context_length: Some(1 << 20),
..Default::default()
};
let profile = EngineProfile {
openai: Some(Arc::new(settings)),
..WireProtocol::default().into()
};
let ctx = build_ctx_with_profile(mock.url.clone(), cfg, profile);
let tool = json!({"type": "function", "function": {"name": "lookup", "parameters": {"type": "object"}}});
let request = json!({
"model": MODEL,
"messages": [{"role": "user", "content": "hi"}],
"tools": [tool],
"stop": "END",
"max_tokens": 8,
});
let ids = ctx.tokenizers.encode_chat(MODEL, &request).unwrap();
send(ctx.clone(), request).await;
let body = captured(&mock);
assert_eq!(body["input_ids"], json!(ids));
assert_eq!(body["require_reasoning"], true);
let sampling = &body["sampling_params"];
assert_eq!(sampling["skip_special_tokens"], false);
assert_eq!(sampling["stop"], json!(["END", "<|close|>tools<|sep|>"]));
assert!(ctx.metrics.render().contains(&format!(
"sgl_router_input_ids_forwarding_total{{model_id=\"{MODEL}\",outcome=\"forwarded\"}} 1"
)));
}
+10 -4
View File
@@ -58,7 +58,7 @@ setting and picks one per request.
select_chat_formatter(&ChatFormatterOptions) -> (Option<ChatFormatter>, Option<String>)
load_chat_formatter(tokenizer_config, model_path, model_type, chat_template) -> Result<ChatFormatter, TemplateError>
ChatFormatter::render_prompt(&dyn OAIChatLikeRequest) -> Result<RenderedPrompt, TemplateError>
ChatFormatter::render_request(serde_json::Value, &default_kwargs) -> Result<(String, String), TemplateError> // DeepSeek-V4, V4.1, GLM-5.3
ChatFormatter::render_request(serde_json::Value, &default_kwargs) -> Result<(RenderedPrompt, String), TemplateError> // DeepSeek-V4, V4.1, GLM-5.3, Kimi K3
DeepSeekV4Profile::from_checkpoint(profile_override, encoder_source) -> Result<DeepSeekV4Profile, String>
ChatFormatter::resolve_thinking(&mut kwargs, tools_enabled, named_tool_choice) -> Option<bool>
ChatFormatter::stop_strs() -> Option<OneOrMany<String>>
@@ -75,7 +75,8 @@ native formatters.
| DeepSeek V4 | `models/deepseek_v4.rs` on Dynamo's V4 encoder |
| DeepSeek V4.1 | `models/deepseek_v41.rs` on Dynamo's V4.1 encoder |
| GLM-5.3, GLM-5.3-Flash | `models/glm53.rs` on the checkpoint's Jinja template, picked by the template |
| DeepSeek V3.2, Kimi K3, Inkling, other Dynamo native models | Dynamo native formatter, with SGLang's thinking defaults |
| Kimi K3 | `models/kimi_k3.rs` on Dynamo's K3 formatter |
| DeepSeek V3.2, Inkling, other Dynamo native models | Dynamo native formatter, with SGLang's thinking defaults |
| Everything else | `load_chat_formatter` |
`load_chat_formatter` (`loader.rs`) follows Python's `template_manager.py`
@@ -93,7 +94,8 @@ with `encode_segments`.
SGLang does. It returns the continuation prefix separately, since
SGLang tokenizes it on its own. `default_kwargs` are the server's
`--default-chat-template-kwargs`, which apply after the request's own values.
On a DeepSeek formatter, `render_prompt` rebuilds the body from the trait.
On a DeepSeek formatter, `render_prompt` rebuilds the body from the trait; on
Kimi K3 it keeps Dynamo's request semantics, which typed hosts rely on.
`from_checkpoint` lets a host that fetches model files itself, such as
sgl-router, build `ChatFormatter::DeepSeekV4` directly. `requested_effort` and
`requested_thinking` (`reasoning.rs`) port `protocol.py`'s `reasoning` handling
@@ -107,6 +109,7 @@ once, for the renderer and DeepSeek-V4 and V4.1 alike. The SGLang additions hook
| `models/deepseek_v41.rs` | the same, `encoding_dsv41.py` | Replaces the OAI formatter. On top of the V4 dump it keeps text parts lists, sends only the request tool fields the client set, and maps effort to the encoder's 1-100 budget, then calls `deepseek::v41::encode_messages`. Developer turns and media are left to the engine. |
| `models/glm53.rs` | `serving_chat.py` (`_process_messages`, `_apply_jinja_template`), `template_detection.py` | Picked by the template (`_is_glm53`). Builds the template context SGLang builds (message and tool dumps, parsed tool arguments, kwargs over the effort, server defaults), then renders it with Dynamo's Jinja formatter, text parts concatenated as the template does. Media, a `chat_template` kwarg, `-0` and U+001C..U+001F in assistant content are left to the engine. |
| `models/kimi_k25.rs` | the Kimi K2.5 checkpoint's tool encoder | Adds `tools_ts_str` (TypeScript tool declarations) to the kwargs before the Jinja render. |
| `models/kimi_k3.rs` | `serving_chat.py` (`_prepare_kimi_k3_messages`, `_encode_messages`), the checkpoint's `encoding_k3.py` | Replaces the OAI formatter's request handling. It prepares the messages, tools, thinking effort, `tool_choice` and `response_format` as SGLang does, then calls Dynamo's `KimiK3Formatter`. Shapes Dynamo renders differently are left to the engine. |
`stop_strs` returns the legacy template's stop strings; Jinja and native
formatters have none, as in Python.
@@ -142,7 +145,10 @@ is the port of SGLang's `BaseReasoningFormatDetector` they configure and
needs no `parser` feature. `tests/reasoning_parity.rs` and the test in
`tool_call/models/mod.rs` check them against SGLang's `ReasoningParser` and
`FunctionCallParser`. To port another model, add `think/models/<model>.rs` or
`tool_call/models/<model>.rs` and its names in that `models/mod.rs`.
`tool_call/models/<model>.rs` and its names in that `models/mod.rs`. A detector
that overrides the base splitting, as Kimi-K3's does, is its own `ThinkConfig`
variant. Kimi-K3's `auto` tool-call structural tag is in `openai/kimi_k3_tag.rs`;
`tests/fixtures/openai_parity/kimi-k3.json` checks it byte for byte.
Pass-through for parsing:
- Other reasoning parsers go through
@@ -0,0 +1,64 @@
//! Kimi-K3's XTML markers and the helpers SGLang's K3 reasoning parser, tool
//! parser and structural tag share (`function_call/kimik3_format.py`).
pub(crate) const THINK_OPEN: &str = "<|open|>think<|sep|>";
pub(crate) const THINK_CLOSE: &str = "<|close|>think<|sep|>";
pub(crate) const RESPONSE_OPEN: &str = "<|open|>response<|sep|>";
pub(crate) const RESPONSE_CLOSE: &str = "<|close|>response<|sep|>";
pub(crate) const TOOLS_OPEN: &str = "<|open|>tools<|sep|>";
pub(crate) const TOOLS_CLOSE: &str = "<|close|>tools<|sep|>";
pub(crate) const MESSAGE_CLOSE: &str = "<|close|>message<|sep|>";
pub(crate) const CALL_OPEN: &str = "<|open|>call";
pub(crate) const CALL_CLOSE: &str = "<|close|>call<|sep|>";
/// `_PARTIAL_MARKER_SUFFIXES`: where `max_tokens` can cut a marker before `<|sep|>`.
const PARTIAL_MARKER_SUFFIXES: [&str; 8] = [
"<|open|>",
"<|close|>",
"<|close|>think",
"<|open|>response",
"<|close|>response",
"<|open|>tools",
"<|close|>tools",
"<|close|>message",
];
/// `partial_suffix_len`: the longest proper marker prefix `text` ends with.
/// Markers are ASCII, so the byte length is a char boundary.
pub(crate) fn partial_suffix_len(text: &str, markers: &[&str]) -> usize {
let mut best = 0;
for marker in markers {
let longest = (marker.len() - 1).min(text.len());
if let Some(len) = (best + 1..=longest)
.rev()
.find(|&len| text.ends_with(&marker[..len]))
{
best = len;
}
}
best
}
/// `strip_partial_marker_suffix`: drop the first matching suffix.
pub(crate) fn strip_partial_marker_suffix(text: &str) -> &str {
PARTIAL_MARKER_SUFFIXES
.iter()
.find_map(|suffix| text.strip_suffix(suffix))
.unwrap_or(text)
}
/// `strip_response_wrappers`: the first response channel's text if there is
/// one, else the text without response closes; message closes dropped.
pub(crate) fn strip_response_wrappers(text: &str) -> String {
let text = match text.find(RESPONSE_OPEN) {
Some(open) => {
let inner = &text[open + RESPONSE_OPEN.len()..];
inner
.find(RESPONSE_CLOSE)
.map_or(inner, |c| &inner[..c])
.to_owned()
}
None => text.replace(RESPONSE_CLOSE, ""),
};
strip_partial_marker_suffix(&text.replace(MESSAGE_CLOSE, "")).to_owned()
}
+2
View File
@@ -6,6 +6,8 @@
//! request types, sampling, validation, transport, and runtime.
mod error;
#[cfg(any(feature = "parser", feature = "openai"))]
mod kimi_k3_format;
mod model_files;
#[cfg(feature = "openai")]
pub mod openai;
+133 -19
View File
@@ -7,6 +7,7 @@ use std::sync::Arc;
use serde::Deserialize;
use serde_json::{Map, Value, json};
use super::kimi_k3_tag::auto_tool_call_tag;
use super::request::{
OneOrList, ResponseFormat, StreamOptions, custom_labels, default_model, float_logit_bias,
format_json_schema, lora_path, one, usage_flags, yes,
@@ -16,6 +17,7 @@ use super::wire::{
stream_error_event, unary_reply,
};
use super::{OpenAiHeaders, OpenAiSettings, OpenAiTokenizer, Unsupported};
use crate::py::python_json;
use crate::think::{ReasoningOptions, ThinkConfig, ThinkDetector, think_config};
use crate::tool_call::{ToolCallItem, ToolDetector, tool_call_id, tool_detector};
@@ -30,8 +32,25 @@ pub struct ChatModel {
/// reasoning config to `special_case="always"`.
#[serde(default)]
pub always_thinks: bool,
/// `chat_encoding_spec`, the encoder SGLang renders the model with; the
/// OpenAI layer special-cases `"kimi_k3"`.
#[serde(default)]
pub chat_encoding_spec: Option<String>,
}
impl ChatModel {
fn is_kimi_k3(&self) -> bool {
self.chat_encoding_spec.as_deref() == Some("kimi_k3")
}
}
/// `KimiK3Detector.eot_token`, which K3 requests with tools also stop on.
const KIMI_K3_TOOLS_CLOSE: &str = "<|close|>tools<|sep|>";
/// `_KIMI_K3_GENERATION_STUB_TOKENS`: K3's assistant generation stub is model
/// input, but not reported as prompt tokens.
const KIMI_K3_GENERATION_STUB_TOKENS: u64 = 3;
/// `ChatCompletionRequest` with Pydantic's defaults, for the fields that reach
/// `GenerateReqInput` or the response. A body this rejects goes to the engine.
#[derive(Deserialize)]
@@ -188,6 +207,8 @@ pub fn lower_chat(
return Err(Unsupported("tool_grammar"));
}
let tools = request_tools(&raw)?;
let tool_definitions = raw.get("tools").and_then(Value::as_array).cloned();
let parallel_tool_calls = raw.get("parallel_tool_calls") != Some(&Value::Bool(false));
let tool_names = match (settings.tool_call_parser.as_deref(), &tools) {
(Some(parser), Some(names)) => {
tool_detector(parser, names.clone()).ok_or(Unsupported("tool_parser"))?;
@@ -229,12 +250,30 @@ pub fn lower_chat(
for (key, value) in settings.default_chat_template_kwargs.iter().flatten() {
kwargs.entry(key.clone()).or_insert_with(|| value.clone());
}
let thinking = parser.is_some()
&& (model.always_thinks || kwargs.get("thinking") == Some(&Value::Bool(true)));
let thinking = match parser.and_then(|config| config.default_thinking(&kwargs)) {
Some(thinking) => thinking,
None => {
parser.is_some()
&& (model.always_thinks || kwargs.get("thinking") == Some(&Value::Bool(true)))
}
};
// `_patch_reasoning_skip_special_tokens`: K3's markers are special tokens.
let kimi_k3 = model.is_kimi_k3();
if kimi_k3 || settings.reasoning_parser.as_deref() == Some("kimi_k3") {
request.skip_special_tokens = false;
}
// `_process_messages`: tool output keeps the parser's special tokens.
if tools.is_some() {
request.skip_special_tokens = false;
}
// K3 also stops on its tool parser's `eot_token`.
let tool_call_stop = match &tool_names {
Some((parser, _)) if kimi_k3 => match parser.as_str() {
"kimi_k3" => Some(KIMI_K3_TOOLS_CLOSE),
_ => return Err(Unsupported("tool_parser")),
},
_ => None,
};
let previous_content = match request.messages.last() {
Some(last)
// Roles are compared lowercased, as Pydantic normalizes them.
@@ -252,9 +291,40 @@ pub fn lower_chat(
_ => None,
};
let lowered = generate_body(
let mut lowered = generate_body(
&request, &kwargs, model, prompt_ids, thinking, headers, settings,
);
// `get_structure_constraint`: K3's tool parser constrains `auto` calls by
// its structural tag, unless the request is already constrained; Python
// tests the constraints by truthiness, so empty strings are unset.
let sampling = &mut lowered["sampling_params"];
if let (Some((tool_parser, _)), Some(definitions)) = (&tool_names, &tool_definitions)
&& tool_parser == "kimi_k3"
&& ["regex", "ebnf", "json_schema"]
.iter()
.all(|key| sampling[*key].as_str().is_none_or(str::is_empty))
{
// `strip_structural_tag_excludes`: the reasoning parser owns the think tokens.
let strip = parser
.map(|config| config.think_tokens())
.unwrap_or_default();
let strip: Vec<&str> = strip.into_iter().filter(|t| !t.is_empty()).collect();
if let Some(tag) = auto_tool_call_tag(definitions, parallel_tool_calls, &strip)? {
sampling["structural_tag"] = python_json(&tag, true).into();
}
}
if let Some(stop) = tool_call_stop {
let sampling = &mut lowered["sampling_params"];
let mut stops = match sampling["stop"].take() {
Value::Array(stops) => stops,
Value::Null => Vec::new(),
single => vec![single],
};
if !stops.iter().any(|s| s == stop) {
stops.push(stop.into());
}
sampling["stop"] = stops.into();
}
let (include_usage, continuous_usage_stats) =
usage_flags(request.stream_options.as_ref(), settings);
let reasoning = parser
@@ -269,8 +339,25 @@ pub fn lower_chat(
== Some(&Value::Bool(true)),
},
});
let history_tool_calls = request
.messages
.iter()
.filter(|m| {
m["role"]
.as_str()
.is_some_and(|r| r.eq_ignore_ascii_case("assistant"))
})
.filter_map(|m| m["tool_calls"].as_array())
.map(|calls| calls.len() as i64)
.sum();
let responder = ChatResponder {
model: request.model,
unreported_prompt_tokens: if kimi_k3 {
KIMI_K3_GENERATION_STUB_TOKENS
} else {
0
},
history_tool_calls,
n: request.n as usize,
logprobs: request.logprobs,
reasoning,
@@ -504,7 +591,7 @@ fn validate(
return invalid;
}
}
format_json_schema(request.response_format.as_ref())?;
format_json_schema(request.response_format.as_ref(), false)?;
Ok(())
}
@@ -634,7 +721,9 @@ fn generate_body(
"sampling_seed": request.seed,
"spaces_between_special_tokens": kwargs.get("spaces_between_special_tokens").cloned().unwrap_or(json!(true)),
});
if let Ok(Some(schema)) = format_json_schema(request.response_format.as_ref()) {
if let Ok(Some(schema)) =
format_json_schema(request.response_format.as_ref(), model.is_kimi_k3())
{
sampling["json_schema"] = schema.into();
}
json!({
@@ -687,6 +776,10 @@ struct ReasoningConfig {
/// Builds the OpenAI chat response from the engine's `/generate` output.
pub struct ChatResponder {
model: String,
/// Prompt tokens `_reported_prompt_tokens` leaves out of usage.
unreported_prompt_tokens: u64,
/// `_get_history_tool_calls_cnt`, which numbers K3 tool-call ids.
history_tool_calls: i64,
n: usize,
logprobs: bool,
reasoning: Option<ReasoningConfig>,
@@ -707,10 +800,13 @@ pub struct ChatResponder {
impl ChatResponder {
/// `_handle_non_streaming_request` over a `/generate` response.
pub fn unary(self, body: &[u8]) -> Option<Reply> {
let ret = match serde_json::from_slice(body).ok()? {
let mut ret = match serde_json::from_slice(body).ok()? {
Value::Array(items) => items,
item => vec![item],
};
for item in &mut ret {
self.report_prompt_tokens(&mut item["meta_info"]);
}
let mut choices = Vec::with_capacity(ret.len());
for (index, item) in ret.iter().enumerate() {
let meta = &item["meta_info"];
@@ -738,7 +834,7 @@ impl ChatResponder {
let calls = calls
.iter()
.enumerate()
.map(|(i, call)| tool_call(i as i64, call));
.map(|(i, call)| self.tool_call(i as i64, call));
tool_calls = calls.collect();
if finish_reason["type"] == "stop" {
finish_reason["type"] = "tool_calls".into();
@@ -792,7 +888,9 @@ impl ChatResponder {
fn stream_frame(&mut self, content: &Value) -> Vec<String> {
let index = content.get("index").and_then(Value::as_u64).unwrap_or(0);
let meta = &content["meta_info"];
self.stream.record(index, meta);
let mut reported = meta.clone();
self.report_prompt_tokens(&mut reported);
self.stream.record(index, &reported);
let finish_reason = meta["finish_reason"].clone();
let finish_type = finish_reason["type"].as_str().map(str::to_owned);
@@ -899,7 +997,7 @@ impl ChatResponder {
}
for call in calls {
self.has_tool_calls.insert(index);
let mut tool_call = tool_call(call.tool_index, &call);
let mut tool_call = self.tool_call(call.tool_index, &call);
// Python streams an id and name only for a non-empty name.
if call.name.is_empty() {
tool_call["id"] = Value::Null;
@@ -965,6 +1063,32 @@ impl ChatResponder {
events
}
/// `_reported_prompt_tokens`.
fn report_prompt_tokens(&self, meta: &mut Value) {
if self.unreported_prompt_tokens > 0 && meta.is_object() {
let prompt = meta_u64(meta, "prompt_tokens");
meta["prompt_tokens"] = prompt.saturating_sub(self.unreported_prompt_tokens).into();
}
}
/// A `ToolCall` as Pydantic dumps it, with `_process_tool_call_id`'s id.
fn tool_call(&self, index: i64, call: &ToolCallItem) -> Value {
let id = match self.tools.as_ref().map(|(parser, _)| parser.as_str()) {
Some("kimi_k3") => format!(
"{}:{}",
call.name,
self.history_tool_calls + call.tool_index
),
_ => tool_call_id(),
};
json!({
"id": id,
"index": index,
"type": "function",
"function": {"name": call.name, "arguments": call.parameters},
})
}
/// `build_sse_content`: msgspec drops a null `usage`, never a null choice field.
fn chunk(
&self,
@@ -1045,16 +1169,6 @@ impl ChatResponder {
}
}
/// A `ToolCall` as Pydantic dumps it; `call_<24 hex>` ids as SGLang mints them.
fn tool_call(index: i64, call: &ToolCallItem) -> Value {
json!({
"id": tool_call_id(),
"index": index,
"type": "function",
"function": {"name": call.name, "arguments": call.parameters},
})
}
#[cfg(test)]
mod tests {
use super::*;
@@ -158,7 +158,7 @@ pub fn lower_completion(
if (request.echo || request.logprobs.is_some()) && tokenizer.is_none() {
return Err(Unsupported("no_tokenizer"));
}
if let Some(schema) = format_json_schema(request.response_format.as_ref())? {
if let Some(schema) = format_json_schema(request.response_format.as_ref(), false)? {
request.json_schema = Some(schema);
}
if request.routed_dp_rank.is_none() {
@@ -0,0 +1,355 @@
//! Kimi-K3's `tool_choice: "auto"` structural tag
//! (`get_kimik3_auto_tool_call_structural_tag` in
//! `function_call/kimik3_structural_tag.py`), dumped as xgrammar's
//! `StructuralTag.model_dump(by_alias=True)`.
use serde_json::{Map, Value, json};
use super::Unsupported;
use crate::kimi_k3_format::{
CALL_CLOSE, CALL_OPEN, THINK_CLOSE, THINK_OPEN, TOOLS_CLOSE, TOOLS_OPEN,
};
const CLOSE_TOKEN: &str = "<|close|>";
const JSON_TYPES: [&str; 7] = [
"string", "number", "integer", "boolean", "array", "object", "null",
];
/// The schema keywords `_schema_types` reads a type from, in its order.
const TYPE_KEYWORDS: [(&str, &str); 4] = [
(
"object",
"additionalProperties dependentRequired dependentSchemas maxProperties minProperties \
patternProperties properties propertyNames required",
),
(
"array",
"contains items maxContains maxItems minContains minItems prefixItems uniqueItems",
),
("string", "format maxLength minLength pattern"),
(
"number",
"exclusiveMaximum exclusiveMinimum maximum minimum multipleOf",
),
];
/// How deep and how long `_schema_types` may run on one argument. Python
/// recurses until its recursion limit and then drops the tag; a schema past
/// these bounds is left to the engine, which does exactly that.
const MAX_SCHEMA_DEPTH: usize = 64;
const MAX_SCHEMA_STEPS: usize = 10_000;
/// The auto tag over `tools` (request tool definitions), without the `strip`
/// tokens `strip_structural_tag_excludes` removes; `None` without tools. Only
/// the trigger's excludes hold think tokens.
pub(super) fn auto_tool_call_tag(
tools: &[Value],
parallel_tool_calls: bool,
strip: &[&str],
) -> Result<Option<Value>, Unsupported> {
if tools.is_empty() {
return Ok(None);
}
let call_tags = tools
.iter()
.map(|tool| tool_call_tag(&tool["function"]))
.collect::<Option<_>>()
.ok_or(Unsupported("tool_definition"))?;
let tools_tag = tool_calls_tag(call_tags, parallel_tool_calls);
// `_auto_suffix`: a retriggered second tools section would evade the single-call limit.
let excludes: Vec<&str> = [THINK_OPEN, THINK_CLOSE, CALL_OPEN]
.into_iter()
.filter(|token| !strip.contains(token))
.collect();
let format = json!({
"type": "triggered_tags",
"triggers": [TOOLS_OPEN],
"tags": [tools_tag],
"at_least_one": false,
"stop_after_first": !parallel_tool_calls,
"excludes": excludes,
});
Ok(Some(json!({"type": "structural_tag", "format": format})))
}
fn escape_attr(value: &str) -> String {
value.replace('&', "&amp;").replace('"', "&quot;")
}
fn any_text(excludes: &[&str]) -> Value {
json!({"type": "any_text", "excludes": excludes, "max_tokens": null, "max_chars": null})
}
fn sequence(elements: Vec<Value>) -> Value {
json!({"type": "sequence", "elements": elements})
}
/// `_tool_call_tag` with `_auto_tool_arguments_format`; `None` past the schema bounds.
fn tool_call_tag(function: &Value) -> Option<Value> {
let name = escape_attr(function["name"].as_str().unwrap_or_default());
Some(json!({
"type": "tag",
"begin": format!("{CALL_OPEN} tool=\"{name}\" index=\""),
"content": sequence(vec![
json!({"type": "regex", "pattern": "[1-9][0-9]*"}),
json!({"type": "const_string", "value": "\"<|sep|>"}),
auto_arguments_format(&function["parameters"])?,
]),
"end": CALL_CLOSE,
}))
}
/// `_tool_calls_tag`.
fn tool_calls_tag(call_tags: Vec<Value>, parallel_tool_calls: bool) -> Value {
let content = if parallel_tool_calls {
json!({"type": "tags_with_separator", "tags": call_tags, "separator": "", "at_least_one": true, "stop_after_first": false})
} else if call_tags.len() == 1 {
call_tags.into_iter().next().expect("one tag")
} else {
json!({"type": "or", "elements": call_tags})
};
json!({"type": "tag", "begin": TOOLS_OPEN, "content": content, "end": TOOLS_CLOSE})
}
/// `_nonempty_argument_format`: a token-based end keeps the first close
/// token out of both content formats.
fn nonempty_argument_format(key: &str, xtml_type: &str) -> Value {
let argument = json!({
"type": "tag",
"begin": format!("<|open|>argument key=\"{}\" type=\"{xtml_type}\"<|sep|>", escape_attr(key)),
"content": sequence(vec![
json!({"type": "exclude_token", "exclude_tokens": []}),
json!({"type": "any_tokens", "exclude_tokens": [], "max_tokens": null}),
]),
"end": {"type": "token", "token": CLOSE_TOKEN},
});
sequence(vec![
argument,
json!({"type": "const_string", "value": "argument<|sep|>"}),
])
}
/// `_auto_tool_arguments_format`: required arguments present and non-empty, then any text.
fn auto_arguments_format(parameters: &Value) -> Option<Value> {
let loose = || Some(any_text(&[]));
let Value::Object(schema) = parameters else {
return loose();
};
let empty = Value::Object(Map::new());
let properties = schema.get("properties").unwrap_or(&empty);
let no_required = Value::Array(Vec::new());
let required = schema.get("required").unwrap_or(&no_required);
let (Value::Object(properties), Value::Array(required)) = (properties, required) else {
return loose();
};
let mut elements = Vec::new();
for key in required {
let Some(key) = key.as_str() else {
return loose();
};
let Some(property) = properties
.get(key)
.filter(|s| s.is_boolean() || s.is_object())
else {
return loose();
};
let Some(xtml_type) = single_xtml_type(property, parameters)? else {
return loose();
};
elements.push(nonempty_argument_format(key, xtml_type));
}
if elements.is_empty() {
let mut alternatives = Vec::new();
for (key, property) in properties {
if (property.is_boolean() || property.is_object())
&& let Some(xtml_type) = single_xtml_type(property, parameters)?
{
alternatives.push(nonempty_argument_format(key, xtml_type));
}
}
match alternatives.len() {
0 => return loose(),
1 => elements.push(alternatives.remove(0)),
_ => elements.push(json!({"type": "or", "elements": alternatives})),
}
}
elements.push(any_text(&[CALL_OPEN, CALL_CLOSE]));
Some(sequence(elements))
}
/// `_single_xtml_type`; the outer `None` is past the schema bounds.
fn single_xtml_type(schema: &Value, root: &Value) -> Option<Option<&'static str>> {
let mut steps = 0;
Some(
match schema_types(schema, root, &[], &mut steps)?.as_slice() {
["integer"] => Some("number"),
[only] => Some(*only),
_ => None,
},
)
}
/// `_json_type`.
fn json_type(value: &Value) -> &'static str {
match value {
Value::Null => "null",
Value::Bool(_) => "boolean",
Value::Number(n) if n.is_f64() => "number",
Value::Number(_) => "integer",
Value::String(_) => "string",
Value::Array(_) => "array",
Value::Object(_) => "object",
}
}
/// `_resolve_local_ref`.
fn resolve_local_ref<'a>(reference: &str, root: &'a Value) -> Option<&'a Value> {
let path = reference.strip_prefix("#/")?;
let mut value = root;
for part in path.split('/') {
let key = part.replace("~1", "/").replace("~0", "~");
value = value.as_object()?.get(&key)?;
}
(value.is_boolean() || value.is_object()).then_some(value)
}
/// The JSON types in `JSON_TYPES` order that are members of `set`.
fn ordered(set: impl Fn(&str) -> bool) -> Vec<&'static str> {
JSON_TYPES.into_iter().filter(|t| set(t)).collect()
}
/// `_schema_types`; `seen` holds the `$ref`s already followed on this path.
/// `None` past `MAX_SCHEMA_DEPTH` or `MAX_SCHEMA_STEPS`.
fn schema_types(
schema: &Value,
root: &Value,
seen: &[&str],
steps: &mut usize,
) -> Option<Vec<&'static str>> {
*steps += 1;
if seen.len() > MAX_SCHEMA_DEPTH || *steps > MAX_SCHEMA_STEPS {
return None;
}
let schema = match schema {
Value::Bool(false) => return Some(Vec::new()),
Value::Object(schema) => schema,
_ => return Some(JSON_TYPES.to_vec()),
};
if let Some(Value::String(reference)) = schema.get("$ref")
&& !seen.contains(&reference.as_str())
&& let Some(target) = resolve_local_ref(reference, root)
{
let seen = [seen, &[reference.as_str()]].concat();
return schema_types(target, root, &seen, steps);
}
// Nesting without `$ref`s also counts toward the depth.
let nested = [seen, &[""]].concat();
let mut branches = |options: &Vec<Value>| -> Option<Vec<Vec<&'static str>>> {
options
.iter()
.filter(|o| o.is_boolean() || o.is_object())
.map(|o| schema_types(o, root, &nested, steps))
.collect()
};
match schema.get("type") {
Some(Value::String(t)) => {
return Some(match JSON_TYPES.iter().find(|j| *j == t) {
Some(t) => vec![*t],
None => JSON_TYPES.to_vec(),
});
}
Some(Value::Array(types)) => return Some(ordered(|t| types.iter().any(|v| v == t))),
_ => {}
}
for keyword in ["anyOf", "oneOf"] {
if let Some(Value::Array(options)) = schema.get(keyword) {
let types = branches(options)?.concat();
return Some(ordered(|t| types.contains(&t)));
}
}
if let Some(Value::Array(options)) = schema.get("allOf") {
let widen = |types: Vec<&'static str>| -> Vec<&'static str> {
let mut types = types;
if types.contains(&"number") && !types.contains(&"integer") {
types.push("integer");
}
types
};
let constrained: Vec<Vec<&'static str>> = branches(options)?
.into_iter()
.filter(|types| ordered(|t| types.contains(&t)) != JSON_TYPES.to_vec())
.collect();
if let Some((first, rest)) = constrained.split_first() {
let mut result = widen(first.clone());
for types in rest {
let widened = widen(types.clone());
result.retain(|t| widened.contains(t));
}
// number survives only if every branch allows it, making integer redundant.
if result.contains(&"number") {
result.retain(|t| *t != "integer");
}
return Some(ordered(|t| result.contains(&t)));
}
}
if let Some(value) = schema.get("const") {
return Some(vec![json_type(value)]);
}
if let Some(Value::Array(values)) = schema.get("enum") {
let types: Vec<&str> = values.iter().map(json_type).collect();
return Some(ordered(|t| types.contains(&t)));
}
Some(
TYPE_KEYWORDS
.iter()
.find(|(_, keywords)| keywords.split_whitespace().any(|k| schema.contains_key(k)))
.map_or_else(|| JSON_TYPES.to_vec(), |(json_type, _)| vec![*json_type]),
)
}
#[cfg(test)]
mod tests {
use super::*;
/// A tool whose required argument is `hops + 1` `$ref`s from its type, or
/// with `fanout`, `hops` levels that each refer twice to the next.
fn ref_tool(hops: usize, fanout: bool) -> Value {
let mut defs: Map<String, Value> = (0..hops)
.map(|i| {
let next = json!({"$ref": format!("#/$defs/D{}", i + 1)});
let def = match fanout {
true => json!({"anyOf": [next, next]}),
false => next,
};
(format!("D{i}"), def)
})
.collect();
defs.insert(format!("D{hops}"), json!({"type": "integer"}));
json!({"type": "function", "function": {"name": "f", "parameters": {
"type": "object", "properties": {"x": {"$ref": "#/$defs/D0"}},
"required": ["x"], "$defs": defs,
}}})
}
/// `$ref` chains and fan-outs past the bounds go to the engine, on a small stack.
#[test]
fn schema_types_are_bounded() {
std::thread::Builder::new()
.stack_size(256 * 1024)
.spawn(|| {
let tag = |tool| auto_tool_call_tag(&[tool], true, &[]);
assert!(matches!(
tag(ref_tool(MAX_SCHEMA_DEPTH - 1, false)),
Ok(Some(_))
));
assert!(tag(ref_tool(MAX_SCHEMA_DEPTH, false)).is_err());
assert!(tag(ref_tool(10_000, false)).is_err());
// Within the depth bound, 2^30 paths exhaust the steps.
assert!(tag(ref_tool(30, true)).is_err());
})
.unwrap()
.join()
.unwrap();
}
}
+1
View File
@@ -8,6 +8,7 @@
mod chat;
mod completions;
mod kimi_k3_tag;
mod pieces;
mod request;
mod wire;
+8 -4
View File
@@ -71,20 +71,24 @@ pub(super) struct JsonSchemaFormat {
}
/// The `json_schema` sampling param a `response_format` sets.
/// `renderer_handles_response_format`: the renderer shows the model a
/// `json_schema` format, so `strict: false` goes unconstrained.
pub(super) fn format_json_schema(
format: Option<&ResponseFormat>,
renderer_handles_response_format: bool,
) -> Result<Option<String>, Unsupported> {
let Some(format) = format else {
return Ok(None);
};
Ok(match format.kind {
FormatKind::JsonSchema => {
let schema = format
.json_schema
.as_ref()
let json_schema = format.json_schema.as_ref();
let schema = json_schema
.and_then(|json_schema| json_schema.schema.clone())
.ok_or(Unsupported("invalid_request"))?;
Some(python_json(&Value::Object(schema), true))
let unconstrained = renderer_handles_response_format
&& json_schema.is_some_and(|json_schema| json_schema.strict == Some(false));
(!unconstrained).then(|| python_json(&Value::Object(schema), true))
}
FormatKind::JsonObject => Some(r#"{"type": "object"}"#.into()),
FormatKind::Text => None,
+366
View File
@@ -1,6 +1,8 @@
//! Python's `json.dumps` and `str.strip`, which SGLang's output text and
//! tool-call arguments go through.
use std::collections::HashMap;
use serde_json::Value;
/// Python's `json.dumps(value, ensure_ascii=ascii)`: `", "` and `": "` separators.
@@ -116,6 +118,339 @@ fn python_float(f: f64) -> String {
format!("{sign}{int}.{frac}")
}
/// About as many containers as Python 3.12's `json.loads` opens before it
/// raises `RecursionError` (its C recursion limit, less the caller's frames).
const MAX_DEPTH: usize = 10_000;
/// `sys.get_int_max_str_digits()`: Python's `int` reads at most this many digits.
pub(crate) const MAX_INT_DIGITS: usize = 4300;
/// `json.dumps(json.loads(text), ensure_ascii=False)`, or `None` where
/// `json.loads` raises. Integers keep their digits, as Python's `int` does; a
/// duplicate key keeps its first position and last value. Lone surrogates,
/// which Python keeps, have no Rust `char` and also give `None`. Neither the
/// scan nor the dump recurses, so nesting costs heap, not stack.
pub(crate) fn python_json_reencode(text: &str) -> Option<String> {
let mut scanner = PyJsonScanner {
text,
pos: 0,
nodes: Vec::new(),
};
let root = scanner.document()?;
Some(scanner.dump(root))
}
/// `json.dumps(dict(pairs), ensure_ascii=False)` over values already dumped: a
/// repeated key keeps its first position and last value.
pub(crate) fn python_dict(pairs: impl IntoIterator<Item = (String, String)>) -> String {
let mut entries: Vec<(String, String)> = Vec::new();
let mut index: HashMap<String, usize> = HashMap::new();
for (key, value) in pairs {
match index.get(&key) {
Some(&i) => entries[i].1 = value,
None => {
index.insert(key.clone(), entries.len());
entries.push((key, value));
}
}
}
let mut out = String::from("{");
for (i, (key, value)) in entries.iter().enumerate() {
if i > 0 {
out.push_str(", ");
}
write_python_str(key, false, &mut out);
out.push_str(": ");
out.push_str(value);
}
out.push('}');
out
}
/// A parsed JSON value; children are indices into the scanner's nodes.
enum Node {
/// A scalar, already dumped.
Leaf(String),
Array(Vec<usize>),
/// Keys in first-seen order, each with its last value, and their positions.
Object(Vec<(String, usize)>, HashMap<String, usize>),
}
/// The scanner of Python's `json` module (`_json.c`).
struct PyJsonScanner<'a> {
text: &'a str,
pos: usize,
nodes: Vec<Node>,
}
impl PyJsonScanner<'_> {
fn rest(&self) -> &str {
&self.text[self.pos..]
}
fn skip_ws(&mut self) {
let rest = self.rest();
self.pos += rest.len() - rest.trim_start_matches([' ', '\t', '\n', '\r']).len();
}
fn eat(&mut self, token: &str) -> bool {
let found = self.rest().starts_with(token);
if found {
self.pos += token.len();
}
found
}
fn push(&mut self, node: Node) -> usize {
self.nodes.push(node);
self.nodes.len() - 1
}
/// The whole text as one value, with the containers still open on a stack.
fn document(&mut self) -> Option<usize> {
// Each open container, with the key an object's next value goes under.
let mut open: Vec<(usize, Option<String>)> = Vec::new();
self.skip_ws();
loop {
let mut node = match self.rest().bytes().next()? {
open_byte @ (b'[' | b'{') => {
if open.len() == MAX_DEPTH {
return None;
}
self.pos += 1;
self.skip_ws();
let is_object = open_byte == b'{';
let id = self.push(match is_object {
true => Node::Object(Vec::new(), HashMap::new()),
false => Node::Array(Vec::new()),
});
if self.eat(if is_object { "}" } else { "]" }) {
id
} else {
let key = if is_object { Some(self.key()?) } else { None };
open.push((id, key));
continue;
}
}
_ => {
let leaf = self.scalar()?;
self.push(Node::Leaf(leaf))
}
};
// Add the value to its container, then close every container it completes.
loop {
let Some(frame) = open.last_mut() else {
self.skip_ws();
return (self.pos == self.text.len()).then_some(node);
};
let (parent, key) = (frame.0, frame.1.take());
let is_object = match (&mut self.nodes[parent], key) {
(Node::Array(items), _) => {
items.push(node);
false
}
(Node::Object(entries, index), Some(key)) => {
match index.get(&key) {
Some(&i) => entries[i].1 = node,
None => {
index.insert(key.clone(), entries.len());
entries.push((key, node));
}
}
true
}
_ => unreachable!("only containers are open, objects with a key"),
};
self.skip_ws();
if self.eat(",") {
self.skip_ws();
if is_object {
open.last_mut().expect("still open").1 = Some(self.key()?);
}
break;
}
if !self.eat(if is_object { "}" } else { "]" }) {
return None;
}
node = open.pop().expect("still open").0;
}
}
}
/// An object key and its `:`.
fn key(&mut self) -> Option<String> {
let key = self.rest().starts_with('"').then(|| self.string())??;
self.skip_ws();
self.eat(":").then_some(())?;
self.skip_ws();
Some(key)
}
/// A literal, string or number, dumped.
fn scalar(&mut self) -> Option<String> {
for token in ["null", "true", "false", "NaN", "Infinity", "-Infinity"] {
if self.eat(token) {
return Some(token.into());
}
}
if self.rest().starts_with('"') {
let mut out = String::new();
write_python_str(&self.string()?, false, &mut out);
return Some(out);
}
self.number()
}
/// `json.dumps` of the value at `root`, walking the nodes with a stack.
fn dump(&self, root: usize) -> String {
let mut out = String::new();
// Each node being written, with how many children it has written.
let mut stack = vec![(root, 0)];
while let Some(&(id, written)) = stack.last() {
let top = stack.len() - 1;
let next = match &self.nodes[id] {
Node::Leaf(text) => {
out.push_str(text);
None
}
Node::Array(items) => {
if written == 0 {
out.push('[');
}
let child = items.get(written).copied();
if child.is_none() {
out.push(']');
} else if written > 0 {
out.push_str(", ");
}
child
}
Node::Object(entries, _) => {
if written == 0 {
out.push('{');
}
match entries.get(written) {
Some((key, child)) => {
if written > 0 {
out.push_str(", ");
}
write_python_str(key, false, &mut out);
out.push_str(": ");
Some(*child)
}
None => {
out.push('}');
None
}
}
}
};
match next {
Some(child) => {
stack[top].1 += 1;
stack.push((child, 0));
}
None => {
stack.pop();
}
}
}
out
}
/// `-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][-+]?[0-9]+)?`: an `int` without a
/// fraction or exponent, else a `float`.
fn number(&mut self) -> Option<String> {
let bytes = self.rest().as_bytes();
let digits = |from: usize| {
bytes
.get(from..)
.map_or(0, |b| b.iter().take_while(|b| b.is_ascii_digit()).count())
};
let mut end = usize::from(bytes.first() == Some(&b'-'));
end += match bytes.get(end) {
Some(b'0') => 1,
Some(b'1'..=b'9') => digits(end),
_ => return None,
};
let int_end = end;
if bytes.get(end) == Some(&b'.') && digits(end + 1) > 0 {
end += 1 + digits(end + 1);
}
if matches!(bytes.get(end), Some(b'e' | b'E')) {
let sign = usize::from(matches!(bytes.get(end + 1), Some(b'+' | b'-')));
let exponent = digits(end + 1 + sign);
if exponent > 0 {
end += 1 + sign + exponent;
}
}
let literal = &self.text[self.pos..self.pos + end];
self.pos += end;
match end == int_end {
true if literal.trim_start_matches('-').len() > MAX_INT_DIGITS => None,
// `int("-0")` is 0.
true if literal == "-0" => Some("0".into()),
true => Some(literal.into()),
false => Some(python_float(literal.parse().ok()?)),
}
}
/// A string literal, as `scanstring` decodes it: raw control characters
/// are rejected and `\u` escapes pair surrogates.
fn string(&mut self) -> Option<String> {
let mut chars = self.rest()[1..].char_indices();
let mut out = String::new();
while let Some((i, c)) = chars.next() {
match c {
'"' => {
self.pos += i + 2;
return Some(out);
}
'\\' => out.push(match chars.next()?.1 {
'"' => '"',
'\\' => '\\',
'/' => '/',
'b' => '\u{8}',
'f' => '\u{c}',
'n' => '\n',
'r' => '\r',
't' => '\t',
'u' => {
let unit = hex4(&mut chars)?;
let code = match unit {
0xd800..=0xdbff => {
let escape = [chars.next()?.1, chars.next()?.1];
let low =
hex4(&mut chars).filter(|u| (0xdc00..=0xdfff).contains(u));
match (escape, low) {
(['\\', 'u'], Some(low)) => {
0x10000 + ((unit - 0xd800) << 10) + (low - 0xdc00)
}
_ => return None,
}
}
unit => unit,
};
char::from_u32(code)?
}
_ => return None,
}),
c if c < ' ' => return None,
c => out.push(c),
}
}
None
}
}
/// Four hex digits of a `\u` escape.
fn hex4(chars: &mut std::str::CharIndices<'_>) -> Option<u32> {
let digits: String = (0..4)
.map(|_| chars.next().map(|(_, c)| c).filter(char::is_ascii_hexdigit))
.collect::<Option<_>>()?;
u32::from_str_radix(&digits, 16).ok()
}
/// Python's `str.isspace`, which also counts the ASCII separators U+001C..U+001F.
fn py_space(c: char) -> bool {
c.is_whitespace() || ('\u{1c}'..='\u{1f}').contains(&c)
@@ -134,6 +469,37 @@ mod tests {
use super::*;
use serde_json::json;
/// Nesting costs heap, not stack: past `MAX_DEPTH`, where Python raises
/// `RecursionError`, the text is not JSON, even on a small stack.
#[test]
fn deep_nesting_is_bounded() {
let nested = |depth: usize| "[".repeat(depth) + &"]".repeat(depth);
std::thread::Builder::new()
.stack_size(64 * 1024)
.spawn(move || {
assert_eq!(
python_json_reencode(&nested(MAX_DEPTH)),
Some(nested(MAX_DEPTH))
);
assert_eq!(python_json_reencode(&nested(MAX_DEPTH + 1)), None);
assert_eq!(python_json_reencode(&nested(1 << 20)), None);
})
.unwrap()
.join()
.unwrap();
}
/// `json.loads` reads integers through `int`, which takes at most 4300 digits.
#[test]
fn long_integers_are_not_json() {
let digits = |n: usize| "1".repeat(n);
let negative = format!("-{}", digits(MAX_INT_DIGITS));
assert_eq!(python_json_reencode(&negative), Some(negative.clone()));
assert_eq!(python_json_reencode(&digits(MAX_INT_DIGITS + 1)), None);
let float = format!("{}.0", digits(MAX_INT_DIGITS + 1));
assert!(python_json_reencode(&float).is_some());
}
#[test]
fn python_json_matches_json_dumps() {
let value = json!({"a": [1, 2.5, 1e-5, 1e16, 100.0, null], "é": "x\"\n\u{1F600}"});
+31 -9
View File
@@ -5,7 +5,10 @@ use std::path::PathBuf;
use dynamo_protocols::types::ChatCompletionRequestMessage;
use dynamo_renderer::deepseek::common::tokens;
use dynamo_renderer::{OAIChatLikeRequest, PromptFormatter, RenderedPrompt, RenderedSegment};
use dynamo_renderer::kimi_k3::KimiK3Formatter;
use dynamo_renderer::{
OAIChatLikeRequest, OAIPromptFormatter, PromptFormatter, RenderedPrompt, RenderedSegment,
};
use serde_json::Value;
use thiserror::Error;
@@ -21,7 +24,7 @@ pub use self::loader::load_chat_formatter;
pub use self::models::DeepSeekV4Profile;
use self::models::{
deep_sort, deepseek_v4_thinking, encode_tools_to_typescript, is_glm53, render_deepseek_v4,
render_deepseek_v41, render_glm53,
render_deepseek_v41, render_glm53, render_kimi_k3,
};
pub use self::reasoning::{requested_effort, requested_thinking};
pub use self::selection::{ChatFormatterOptions, select_chat_formatter};
@@ -50,6 +53,9 @@ pub enum ChatFormatter {
DeepSeekV41,
/// GLM-5.3's Jinja template, which `render_request` renders as SGLang does.
Glm53(PromptFormatter),
/// Dynamo's K3 formatter: [`Self::render_prompt`] keeps Dynamo's request
/// semantics, while [`Self::render_request`] follows SGLang's.
KimiK3,
Legacy(Box<LegacyFormatter>),
}
@@ -87,6 +93,13 @@ impl ChatFormatter {
}
render_oai(formatter, &TemplateArgsRequest { request, args })
}
ChatFormatter::KimiK3 => {
KimiK3Formatter::new(true)
.render_prompt(request)
.map_err(|error| TemplateError::Renderer {
message: error.to_string(),
})
}
ChatFormatter::DeepSeekV4(_) | ChatFormatter::DeepSeekV41 => {
// Typed messages serialize straight to JSON, skipping minijinja.
let messages = match request.typed_messages() {
@@ -104,6 +117,7 @@ impl ChatFormatter {
});
body["messages"] = messages;
let (prompt, prefix) = self.render_request(body, &HashMap::new())?;
let prompt = prompt.into_text();
// SGLang tokenizes the prefix on its own and drops its leading BOS.
let prefix = prefix.strip_prefix(tokens::BOS).unwrap_or(&prefix);
if prefix.is_empty() {
@@ -122,19 +136,23 @@ impl ChatFormatter {
/// `chat_template_kwargs`, returning the prompt and the
/// `continue_final_message` prefix SGLang tokenizes separately.
/// The body must be one SGLang's `ChatCompletionRequest` accepts; it is not
/// re-validated. Only DeepSeek-V4, V4.1 and GLM-5.3 render this way; others use
/// [`Self::render_prompt`].
/// re-validated. Only DeepSeek-V4, V4.1, GLM-5.3 and Kimi K3 render this way;
/// others use [`Self::render_prompt`].
pub fn render_request(
&self,
request: Value,
default_kwargs: &HashMap<String, Value>,
) -> Result<(String, String), TemplateError> {
) -> Result<(RenderedPrompt, String), TemplateError> {
let text = |(prompt, prefix)| (RenderedPrompt::text(prompt), prefix);
match self {
ChatFormatter::DeepSeekV4(profile) => {
render_deepseek_v4(*profile, request, default_kwargs)
render_deepseek_v4(*profile, request, default_kwargs).map(text)
}
ChatFormatter::DeepSeekV41 => render_deepseek_v41(request, default_kwargs),
ChatFormatter::Glm53(formatter) => render_glm53(formatter, request, default_kwargs),
ChatFormatter::DeepSeekV41 => render_deepseek_v41(request, default_kwargs).map(text),
ChatFormatter::Glm53(formatter) => {
render_glm53(formatter, request, default_kwargs).map(text)
}
ChatFormatter::KimiK3 => render_kimi_k3(request, default_kwargs),
_ => Err("this formatter renders through render_prompt".into()),
}
.map_err(|message| TemplateError::Renderer { message })
@@ -150,7 +168,8 @@ impl ChatFormatter {
| ChatFormatter::KimiK25 { .. }
| ChatFormatter::DeepSeekV4(_)
| ChatFormatter::DeepSeekV41
| ChatFormatter::Glm53(_) => None,
| ChatFormatter::Glm53(_)
| ChatFormatter::KimiK3 => None,
ChatFormatter::Legacy(formatter) => formatter.spec.stop_str.clone(),
}
}
@@ -174,6 +193,9 @@ impl ChatFormatter {
| ChatFormatter::KimiK25 { thinking, .. } => thinking
.for_request(tools_enabled)
.apply(args, named_tool_choice),
ChatFormatter::KimiK3 => ThinkingTemplates::native(true, true)
.for_request(tools_enabled)
.apply(args, named_tool_choice),
ChatFormatter::DeepSeekV4(_) | ChatFormatter::DeepSeekV41 => {
let enabled = deepseek_v4_thinking(
&serde_json::json!({ "chat_template_kwargs": args }),
@@ -254,7 +254,7 @@ pub(super) fn normalize_messages(messages: &mut [Value]) -> Result<(), String> {
}
/// Each assistant tool call's `function.arguments`, absent ones as `null`.
fn tool_arguments(message: &mut Value) -> impl Iterator<Item = &mut Value> {
pub(super) fn tool_arguments(message: &mut Value) -> impl Iterator<Item = &mut Value> {
let is_assistant = message["role"] == "assistant";
message
.get_mut("tool_calls")
@@ -0,0 +1,364 @@
//! Kimi K3 prompts as SGLang's `serving_chat.py` builds them, on Dynamo's K3 formatter.
use std::collections::HashMap;
use dynamo_renderer::kimi_k3::KimiK3Formatter;
use dynamo_renderer::{OAIChatLikeRequest, OAIPromptFormatter, RenderedPrompt};
use serde_json::{Map, Value, json};
use super::deepseek_v4::{engine_message, pydantic_bool, tool_arguments};
use super::schema_types::normalize_json_schema_types;
use crate::render::reasoning::{requested_effort, requested_thinking};
const IMAGE_PLACEHOLDER: &str = "<|kimi_image_placeholder|>";
const THINKING_EFFORTS: [&str; 3] = ["low", "high", "max"];
const TOKENIZER_ARGS: [&str; 8] = [
"conversation",
"messages",
"tools",
"add_generation_prompt",
"padding",
"truncation",
"max_length",
"return_tensors",
];
/// Render an SGLang chat request body. Returns the prompt, whose segments keep
/// the special-token boundaries, and the `continue_final_message` prefix,
/// which SGLang tokenizes separately.
pub(crate) fn render(
mut request: Value,
defaults: &HashMap<String, Value>,
) -> Result<(RenderedPrompt, String), String> {
let Some(Value::Array(messages)) = request.get_mut("messages").map(Value::take) else {
return Err("messages must be an array".into());
};
let mut messages = messages
.into_iter()
.map(|message| prepare_message(engine_message(message)))
.collect::<Result<Vec<_>, _>>()?;
let request = &request;
// `_handle_last_assistant_message`, which K3 runs only to continue the turn.
let mut prefix = String::new();
if request
.get("continue_final_message")
.map_or(Ok(false), pydantic_bool)?
&& let Some(last) = messages.last().filter(|m| m["role"] == "assistant")
&& let Some(content) = last["content"].as_str()
{
prefix = content.to_owned();
messages.pop();
}
// The template kwargs: the request's, with `reasoning_effort` moved to the
// request effort, then what `reasoning` implies and the server's defaults.
let mut kwargs = request["chat_template_kwargs"]
.as_object()
.cloned()
.unwrap_or_default();
let kwargs_effort = kwargs.remove("reasoning_effort").filter(|e| !e.is_null());
if let Some(thinking) = requested_thinking(request) {
kwargs.entry("thinking").or_insert(thinking.into());
}
for (key, value) in defaults {
kwargs.entry(key).or_insert_with(|| value.clone());
}
// Kimi's `apply_chat_template` and `build_chat_segments` take these as their
// own arguments: SGLang truncates or pads the ids, or fails on a duplicate argument.
if TOKENIZER_ARGS.iter().any(|arg| kwargs.contains_key(*arg)) {
return Err(
"Kimi K3 tokenizer arguments in chat_template_kwargs render on the engine".into(),
);
}
let thinking = kwargs
.get("thinking")
.is_none_or(|thinking| minijinja::Value::from_serialize(thinking).is_true());
let supported = |effort: &Value| {
effort
.as_str()
.is_some_and(|effort| THINKING_EFFORTS.contains(&effort))
};
let effort = kwargs_effort
.as_ref()
.or_else(|| requested_effort(request))
.or_else(|| defaults.get("reasoning_effort").filter(|e| !e.is_null()));
// The tokenizer's default is `max`.
let thinking_effort = match kwargs.get("thinking_effort") {
Some(thinking_effort) => thinking_effort.clone(),
None => effort
.filter(|effort| supported(effort))
.cloned()
.unwrap_or_else(|| "max".into()),
};
// `build_chat_segments` rejects other efforts; Dynamo cannot leave out a null one.
if thinking && !supported(&thinking_effort) {
return Err(format!("unsupported thinking_effort {thinking_effort}"));
}
// Dynamo drops whitespace-only reasoning, which SGLang renders.
if thinking
&& messages.iter().any(|m| {
m["role"] == "assistant"
&& m["reasoning_content"]
.as_str()
.is_some_and(|r| !r.is_empty() && r.trim().is_empty())
})
{
return Err("whitespace-only Kimi K3 reasoning renders on the engine".into());
}
let tools = match request["tools"].as_array() {
Some(tools) if !tools.is_empty() => {
Some(tools.iter().map(tool_dump).collect::<Result<Value, _>>()?)
}
_ => None,
};
// `_effective_tools`: the request's and system turns' declarations.
let has_tools = tools.is_some() || messages.iter().any(|m| m.get("tools").is_some());
// `_validate_request` rejects a required choice without tools.
if !has_tools
&& request["tool_choice"]
.as_str()
.is_some_and(|choice| choice.eq_ignore_ascii_case("required"))
{
return Err("tool_choice required needs tools".into());
}
if has_tools && matches!(request["tool_choice"].as_str(), Some("required" | "none")) {
kwargs
.entry("tool_choice")
.or_insert_with(|| request["tool_choice"].clone());
}
let request_format = response_format_dump(&request["response_format"]);
// `_validate_request` rejects a `json_schema` format without a schema.
if request_format
.as_ref()
.is_some_and(|f| f["type"] == "json_schema" && f["json_schema"]["schema"].is_null())
{
return Err("json_schema response format requires a schema".into());
}
// The encoder also takes a bare type name.
let mut response_format = match kwargs.get("response_format") {
Some(Value::String(kind)) => Some(json!({ "type": kind })),
Some(format) => Some(format.clone()),
None => request_format,
};
if let (Some(format), Some(schema)) = (
response_format.as_mut().and_then(Value::as_object_mut),
kwargs.get("response_schema"),
) {
format.insert("json_schema".into(), json!({ "schema": schema }));
}
let encoder_input = KimiRequest {
messages,
tools,
// The encoder renders only these choices; a named one stays a constraint.
tool_choice: kwargs
.get("tool_choice")
.filter(|choice| matches!(choice.as_str(), Some("required" | "none")))
.cloned(),
response_format,
args: HashMap::from([
("thinking".into(), thinking.into()),
("thinking_effort".into(), thinking_effort),
]),
};
if (encoder_input.messages.iter())
.chain(&encoder_input.tools)
.chain(&encoder_input.response_format)
.any(prints_unlike_python)
{
return Err("Kimi K3 numbers serde_json prints unlike Python render on the engine".into());
}
// SGLang renders every declared tool, whatever the tool_choice.
let prompt = KimiK3Formatter::new(false)
.render_prompt(&encoder_input)
.map_err(|error| error.to_string())?;
Ok((prompt, prefix))
}
/// `_prepare_kimi_k3_messages` on one dumped message, after
/// `normalize_assistant_tool_call_arguments(strict=False)`.
fn prepare_message(mut message: Value) -> Result<Value, String> {
if let Some(parts) = message["content"].as_array_mut() {
// Media expands into image tokens only on the engine.
if parts.iter().any(|part| {
matches!(
part["type"].as_str(),
Some("image_url" | "video_url" | "audio_url" | "input_audio")
)
}) {
return Err("Kimi K3 media renders on the engine".into());
}
parts.retain(|part| part["type"] == "text");
}
neutralize(&mut message["content"]);
if let Some(reasoning) = message.get_mut("reasoning_content") {
neutralize(reasoning);
}
for arguments in tool_arguments(&mut message) {
if let Some(object) = arguments
.as_str()
.and_then(|text| serde_json::from_str::<Value>(text).ok())
.filter(Value::is_object)
{
*arguments = object;
}
// SGLang's encoder and Dynamo parse other text differently.
if arguments.as_str().is_some_and(|text| !text.is_empty()) {
return Err("non-object tool arguments render on the engine".into());
}
neutralize(arguments);
}
if message["role"] == "developer" {
message["role"] = "system".into();
}
// A system turn with tools renders only its declaration; Dynamo rejects
// tools on other turns, which SGLang ignores.
let tools = message.as_object_mut().and_then(|m| m.remove("tools"));
if let Some(Value::Array(tools)) = tools.filter(|_| message["role"] == "system")
&& !tools.is_empty()
{
message["tools"] = tools.iter().map(tool_dump).collect::<Result<_, _>>()?;
message["content"] = "".into();
}
Ok(message)
}
/// `neutralize_kimi_k3_image_placeholder_value`: text cannot spell the image placeholder.
fn neutralize(value: &mut Value) {
match value {
Value::String(text) if text.contains(IMAGE_PLACEHOLDER) => {
*text = text.replace(IMAGE_PLACEHOLDER, "<| kimi_image_placeholder |>");
}
Value::Array(items) => items.iter_mut().for_each(neutralize),
Value::Object(fields) => fields.values_mut().for_each(neutralize),
_ => {}
}
}
/// Whether `value` holds a number Dynamo's serde_json prints unlike Python's
/// `json.dumps`: floats in exponent notation (`1e-06`, `1e+16`), integers
/// beyond 64 bits, and negative zero, which serde_json also reads from `-0`.
fn prints_unlike_python(value: &Value) -> bool {
match value {
Value::Number(number) => number.as_f64().is_some_and(|f| {
let positive_zero = f == 0.0 && f.is_sign_positive();
number.is_f64() && !positive_zero && !(1e-4..1e16).contains(&f.abs())
}),
Value::Array(items) => items.iter().any(prints_unlike_python),
Value::Object(fields) => fields.values().any(prints_unlike_python),
_ => false,
}
}
/// `Tool.model_dump(exclude_unset=True)`: the fields the client set, coerced,
/// after `_validate_request` normalizes the schema types. The encoder sorts the keys.
fn tool_dump(tool: &Value) -> Result<Value, String> {
let f = &tool["function"];
f["name"].as_str().ok_or("tool function requires name")?;
let mut function: Map<String, Value> = ["name", "description", "parameters"]
.into_iter()
.filter_map(|key| Some((key.into(), f.get(key)?.clone())))
.collect();
if let Some(parameters) = function.get_mut("parameters") {
normalize_json_schema_types(parameters);
}
if let Some(strict) = f.get("strict") {
function.insert("strict".into(), pydantic_bool(strict)?.into());
}
// `Tool._propagate_defer_loading`; `Function` drops an unset flag.
if let Some(defer) = [f.get("defer_loading"), tool.get("defer_loading")]
.into_iter()
.flatten()
.find(|v| !v.is_null())
{
function.insert("defer_loading".into(), pydantic_bool(defer)?.into());
}
let mut dump = Map::new();
if let Some(kind) = tool.get("type") {
dump.insert("type".into(), kind.clone());
}
dump.insert("function".into(), function.into());
if let Some(defer) = tool.get("defer_loading") {
let defer = match defer {
Value::Null => Value::Null,
defer => pydantic_bool(defer)?.into(),
};
dump.insert("defer_loading".into(), defer);
}
Ok(dump.into())
}
/// `ResponseFormat.model_dump(exclude_unset=True)`, after `set_json_schema`
/// lifts a legacy top-level `schema` without its `strict` property.
fn response_format_dump(format: &Value) -> Option<Value> {
let kind = format.get("type")?;
let json_schema = match (format.get("json_schema"), format.get("schema")) {
(Some(Value::Object(json_schema)), _) if !json_schema.is_empty() => Some(
json_schema
.iter()
.filter(|(key, _)| {
["name", "description", "schema", "strict"].contains(&key.as_str())
})
.map(|(key, value)| (key.clone(), value.clone()))
.collect::<Map<_, _>>()
.into(),
),
(_, Some(Value::Object(schema))) if kind == "json_schema" && !schema.is_empty() => {
let mut schema = schema.clone();
if let Some(Value::Object(properties)) = schema.get_mut("properties") {
properties.remove("strict");
}
Some(json!({ "schema": schema }))
}
_ => None,
};
let mut dump = json!({ "type": kind });
if let Some(json_schema) = json_schema {
dump["json_schema"] = json_schema;
}
Some(dump)
}
/// The request Dynamo's formatter sees, built the way SGLang calls its encoder.
struct KimiRequest {
messages: Vec<Value>,
tools: Option<Value>,
tool_choice: Option<Value>,
response_format: Option<Value>,
args: HashMap<String, Value>,
}
impl OAIChatLikeRequest for KimiRequest {
fn model(&self) -> String {
String::new()
}
fn messages(&self) -> minijinja::Value {
minijinja::Value::from_serialize(&self.messages)
}
fn tools(&self) -> Option<minijinja::Value> {
self.tools.as_ref().map(minijinja::Value::from_serialize)
}
fn tool_choice(&self) -> Option<minijinja::Value> {
self.tool_choice
.as_ref()
.map(minijinja::Value::from_serialize)
}
fn response_format(&self) -> Option<minijinja::Value> {
self.response_format
.as_ref()
.map(minijinja::Value::from_serialize)
}
fn should_add_generation_prompt(&self) -> bool {
true
}
fn chat_template_args(&self) -> Option<&HashMap<String, Value>> {
Some(&self.args)
}
}
@@ -4,6 +4,7 @@ mod deepseek_v4;
mod deepseek_v41;
mod glm53;
mod kimi_k25;
mod kimi_k3;
mod schema_types;
pub use self::deepseek_v4::DeepSeekV4Profile;
@@ -12,4 +13,5 @@ pub(super) use self::deepseek_v4::{
};
pub(super) use self::deepseek_v41::render as render_deepseek_v41;
pub(super) use self::glm53::{is_glm53, render as render_glm53};
pub(super) use self::kimi_k3::render as render_kimi_k3;
pub(super) use self::kimi_k25::{deep_sort, encode_tools_to_typescript};
@@ -76,15 +76,8 @@ pub fn select_chat_formatter(
None,
);
}
if let Some(formatter) = kimi_k3_formatter_for(&model_type_lower, &display_name_lower, true)
{
return (
Some(ChatFormatter::HuggingFace {
formatter,
thinking: ThinkingTemplates::native(true, true),
}),
None,
);
if kimi_k3_formatter_for(&model_type_lower, &display_name_lower, true).is_some() {
return (Some(ChatFormatter::KimiK3), None);
}
if model_type_lower.as_deref() == Some("inkling_mm_model")
&& let Some(formatter) = native_formatter_for(&model_type_lower, &display_name_lower)
+81 -4
View File
@@ -4,7 +4,13 @@
mod models;
#[cfg(feature = "openai")]
use serde_json::{Map, Value};
use self::models::KimiK3Detector;
pub(crate) use self::models::think_config;
#[cfg(feature = "openai")]
use crate::kimi_k3_format;
/// The request settings Python passes its reasoning parser. Dynamo-backed
/// names only honour `force_reasoning`; `None` keeps the parser's default.
@@ -28,9 +34,80 @@ impl Default for ReasoningOptions {
}
}
/// The detector one `--reasoning-parser` name selects.
#[derive(Debug, Clone, Copy)]
pub(crate) enum ThinkConfig {
/// `BaseReasoningFormatDetector` over these tokens.
Tags(TagConfig),
/// `KimiK3Detector`, which overrides the base splitting.
KimiK3,
}
#[cfg(feature = "openai")]
impl ThinkConfig {
/// `_get_reasoning_from_request` for a template without a reasoning
/// config, by the detector's `reasoning_default`. `None` for `"always"`,
/// which the host resolves from the template instead.
pub(crate) fn default_thinking(&self, kwargs: &Map<String, Value>) -> Option<bool> {
match self {
ThinkConfig::Tags(_) => None,
// `"thinking"`: on unless the merged kwargs turn `thinking` off.
ThinkConfig::KimiK3 => Some(kwargs.get("thinking") != Some(&Value::Bool(false))),
}
}
/// The detector's `think_start_token` and `think_end_token`.
pub(crate) fn think_tokens(&self) -> [&'static str; 2] {
match self {
ThinkConfig::Tags(tags) => [tags.start, tags.end],
ThinkConfig::KimiK3 => [kimi_k3_format::THINK_OPEN, kimi_k3_format::THINK_CLOSE],
}
}
}
/// Streaming and one-shot reasoning splitting for one choice.
#[derive(Debug, Clone)]
pub(crate) enum ThinkDetector {
Tags(TagDetector),
KimiK3(KimiK3Detector),
}
impl ThinkDetector {
pub(crate) fn new(config: ThinkConfig, options: &ReasoningOptions) -> Self {
match config {
ThinkConfig::Tags(tags) => Self::Tags(TagDetector::new(tags, options)),
ThinkConfig::KimiK3 => Self::KimiK3(KimiK3Detector::new(options)),
}
}
/// `detect_and_parse`: `(reasoning_text, normal_text)` of a whole output.
pub(crate) fn parse(&mut self, text: &str) -> (String, String) {
match self {
Self::Tags(detector) => detector.parse(text),
Self::KimiK3(detector) => detector.parse(text),
}
}
/// `parse_streaming_increment`: the `(reasoning, normal)` delta of one chunk.
pub(crate) fn push(&mut self, text: &str) -> (String, String) {
match self {
Self::Tags(detector) => detector.push(text),
Self::KimiK3(detector) => detector.push(text),
}
}
/// `finish`: flush what the stream still holds when it ends.
pub(crate) fn finish(&mut self) -> (String, String) {
match self {
Self::Tags(detector) => detector.finish(),
Self::KimiK3(detector) => detector.finish(),
}
}
}
/// The tokens one `--reasoning-parser` configures the base detector with.
#[derive(Debug, Clone, Copy)]
pub(crate) struct ThinkConfig {
pub(crate) struct TagConfig {
pub start: &'static str,
pub end: &'static str,
pub tool_start: Option<&'static str>,
@@ -39,7 +116,7 @@ pub(crate) struct ThinkConfig {
/// Streaming and one-shot `<think>` splitting, as SGLang's base detector does it.
#[derive(Debug, Clone)]
pub(crate) struct ThinkDetector {
pub(crate) struct TagDetector {
start: &'static str,
end: &'static str,
tool_start: Option<&'static str>,
@@ -54,8 +131,8 @@ pub(crate) struct ThinkDetector {
accumulated_reasoning: String,
}
impl ThinkDetector {
pub(crate) fn new(config: ThinkConfig, options: &ReasoningOptions) -> Self {
impl TagDetector {
pub(crate) fn new(config: TagConfig, options: &ReasoningOptions) -> Self {
let previous_content = options.previous_content.clone().unwrap_or_default();
let mut in_reasoning = options.force_reasoning.unwrap_or(false);
if previous_content.contains(config.start) {
@@ -1,9 +1,9 @@
//! DeepSeek-V4 and V4.1 (`DeepSeekV4Detector` and `DeepSeekV41ReasoningDetector`
//! in `parser/reasoning_parser.py`).
use crate::think::ThinkConfig;
use crate::think::TagConfig;
pub(super) const THINK: ThinkConfig = ThinkConfig {
pub(super) const THINK: TagConfig = TagConfig {
start: "<think>",
end: "</think>",
tool_start: Some("<|DSML|tool_calls>"),
@@ -11,7 +11,7 @@ pub(super) const THINK: ThinkConfig = ThinkConfig {
};
/// V4.1's tool-call anchor is the spaced `<|DSML| calls>` tag.
pub(super) const THINK_V41: ThinkConfig = ThinkConfig {
pub(super) const THINK_V41: TagConfig = TagConfig {
tool_start: Some("<|DSML| calls>"),
..THINK
};
@@ -1,9 +1,9 @@
//! GLM-4.5 through GLM-5.3 (`Glm45Detector` in `parser/reasoning_parser.py`):
//! a `<tool_call>` anywhere also ends the reasoning.
use crate::think::ThinkConfig;
use crate::think::TagConfig;
pub(super) const THINK: ThinkConfig = ThinkConfig {
pub(super) const THINK: TagConfig = TagConfig {
start: "<think>",
end: "</think>",
tool_start: Some("<tool_call>"),
@@ -0,0 +1,280 @@
//! Kimi-K3 (`KimiK3Detector` in `parser/reasoning_parser.py`): the XTML
//! think channel, with post-reasoning content unwrapped from the `response`
//! and `message` markers and a `tools` channel passed through raw.
use crate::kimi_k3_format::{
MESSAGE_CLOSE, RESPONSE_CLOSE, RESPONSE_OPEN, THINK_CLOSE, THINK_OPEN, TOOLS_OPEN,
partial_suffix_len, strip_partial_marker_suffix, strip_response_wrappers,
};
use crate::think::ReasoningOptions;
#[derive(Debug, Clone)]
pub(crate) struct KimiK3Detector {
in_reasoning: bool,
stream_reasoning: bool,
force_nonempty_content: bool,
buffer: String,
stripped_think_start: bool,
reasoning_done: bool,
tools_passthrough: bool,
stream_text: String,
streamed_reasoning: Vec<String>,
discard_delayed_think_close: bool,
}
impl KimiK3Detector {
pub(crate) fn new(options: &ReasoningOptions) -> Self {
// K3 forces reasoning unless the request says otherwise.
let mut in_reasoning = options.force_reasoning.unwrap_or(true);
let previous = options.previous_content.as_deref().unwrap_or_default();
if previous.contains(THINK_OPEN) {
in_reasoning = true;
}
if previous.contains(THINK_CLOSE) {
in_reasoning = false;
}
Self {
in_reasoning,
stream_reasoning: options.stream_reasoning,
force_nonempty_content: options.force_nonempty_content,
buffer: String::new(),
stripped_think_start: false,
reasoning_done: false,
tools_passthrough: false,
stream_text: String::new(),
streamed_reasoning: Vec::new(),
discard_delayed_think_close: false,
}
}
/// `detect_and_parse`.
pub(crate) fn parse(&mut self, text: &str) -> (String, String) {
let in_reasoning = self.in_reasoning || text.contains(THINK_OPEN);
if !in_reasoning && !text.contains(THINK_CLOSE) {
return (String::new(), clean_content(text));
}
if self.force_nonempty_content && is_skipped_think_answer(text) {
return (String::new(), clean_content(text));
}
let start = text.find(THINK_OPEN).map_or(0, |i| i + THINK_OPEN.len());
let close_idx = find_from(text, THINK_CLOSE, start);
let tools_idx = find_from(text, TOOLS_OPEN, start);
let channel_idx = next_channel_idx(text, start);
if skipped_think_channel(text, start, close_idx, channel_idx) {
let skipped = text[start..].replace(THINK_CLOSE, "");
return (String::new(), clean_content(&skipped));
}
if let (Some(close), Some(tools)) = (close_idx, tools_idx)
&& tools < close
{
return (
strip_partial_marker_suffix(&text[start..tools]).to_owned(),
clean_content(&text[tools..]),
);
}
let Some(close) = close_idx else {
if let Some(channel) = channel_idx {
return (
strip_partial_marker_suffix(&text[start..channel]).to_owned(),
clean_content(&text[channel..]),
);
}
return (
strip_partial_marker_suffix(&text[start..]).to_owned(),
String::new(),
);
};
let rest = &text[close + THINK_CLOSE.len()..];
(text[start..close].to_owned(), clean_content(rest))
}
/// `parse_streaming_increment`.
pub(crate) fn push(&mut self, text: &str) -> (String, String) {
self.buffer.push_str(text);
if self.force_nonempty_content {
self.stream_text.push_str(text);
}
if !self.in_reasoning && !self.reasoning_done {
if let Some(open) = self.buffer.find(THINK_OPEN) {
self.buffer.drain(..open + THINK_OPEN.len());
self.in_reasoning = true;
self.stripped_think_start = true;
} else if THINK_OPEN.starts_with(self.buffer.as_str()) {
return Default::default();
} else {
self.reasoning_done = true;
}
}
if !self.in_reasoning {
return (String::new(), self.drain_content());
}
if !self.stripped_think_start
&& let Some(open) = self.buffer.find(THINK_OPEN)
{
self.buffer.drain(..open + THINK_OPEN.len());
self.stripped_think_start = true;
}
let buf = self.buffer.clone();
let close_idx = buf.find(THINK_CLOSE);
let tools_idx = buf.find(TOOLS_OPEN);
let channel_idx = next_channel_idx(&buf, 0);
if skipped_think_channel(&buf, 0, close_idx, channel_idx) {
let replay = std::mem::take(&mut self.streamed_reasoning).concat();
self.in_reasoning = false;
self.reasoning_done = true;
self.discard_delayed_think_close = true;
return (String::new(), replay + &self.drain_content());
}
if let Some(close) = close_idx
&& tools_idx.is_none_or(|tools| tools >= close)
{
self.buffer = buf[close + THINK_CLOSE.len()..].to_owned();
self.in_reasoning = false;
self.reasoning_done = true;
self.streamed_reasoning.clear();
return (buf[..close].to_owned(), self.drain_content());
}
if let Some(channel) = channel_idx {
let reasoning = strip_partial_marker_suffix(&buf[..channel]).to_owned();
self.buffer = buf[channel..].to_owned();
self.in_reasoning = false;
self.reasoning_done = true;
self.streamed_reasoning.clear();
self.tools_passthrough = buf[channel..].starts_with(TOOLS_OPEN);
return (reasoning, self.drain_content());
}
if !self.stream_reasoning {
return Default::default();
}
let mut markers = vec![
THINK_CLOSE,
TOOLS_OPEN,
RESPONSE_OPEN,
RESPONSE_CLOSE,
MESSAGE_CLOSE,
];
if !self.stripped_think_start {
markers.push(THINK_OPEN);
}
let holdback = partial_suffix_len(&buf, &markers);
let emit = strip_partial_marker_suffix(&buf[..buf.len() - holdback]).to_owned();
self.buffer = buf[emit.len()..].to_owned();
self.streamed_reasoning.push(emit.clone());
(emit, String::new())
}
/// `finish`.
pub(crate) fn finish(&mut self) -> (String, String) {
self.streamed_reasoning.clear();
if !self.force_nonempty_content {
return self.base_finish();
}
let text = std::mem::take(&mut self.stream_text);
if self.in_reasoning && is_skipped_think_answer(&text) {
self.buffer.clear();
return (String::new(), clean_content(&text));
}
if self.in_reasoning && !self.stream_reasoning && !self.buffer.is_empty() {
return (std::mem::take(&mut self.buffer), String::new());
}
Default::default()
}
/// `BaseReasoningFormatDetector.finish` without `force_nonempty_content`.
fn base_finish(&mut self) -> (String, String) {
let buffer = std::mem::take(&mut self.buffer);
if !self.in_reasoning {
return (String::new(), buffer);
}
let buffer = buffer.strip_prefix(THINK_OPEN).unwrap_or(&buffer);
(buffer.to_owned(), String::new())
}
/// `_drain_content`: the buffered content, unwrapped, holding back partial markers.
fn drain_content(&mut self) -> String {
if self.buffer.is_empty() {
return String::new();
}
let buf = std::mem::take(&mut self.buffer);
let delayed_close = |buf: &str| match self.discard_delayed_think_close {
true => partial_suffix_len(buf, &[THINK_CLOSE]),
false => 0,
};
let discard = |text: &str| match self.discard_delayed_think_close {
true => text.replace(THINK_CLOSE, ""),
false => text.to_owned(),
};
if self.tools_passthrough {
let (emit, held) = buf.split_at(buf.len() - delayed_close(&buf));
self.buffer = held.to_owned();
return discard(emit);
}
if let Some(tools) = buf.find(TOOLS_OPEN) {
let (emit, held) = buf.split_at(buf.len() - delayed_close(&buf));
self.buffer = held.to_owned();
let (head, tail) = emit.split_at(tools.min(emit.len()));
let head = discard(&strip_markers(head));
self.tools_passthrough = true;
return head + &discard(tail);
}
let mut markers = vec![RESPONSE_OPEN, RESPONSE_CLOSE, MESSAGE_CLOSE, TOOLS_OPEN];
if self.discard_delayed_think_close {
markers.push(THINK_CLOSE);
}
let (emit, held) = buf.split_at(buf.len() - partial_suffix_len(&buf, &markers));
self.buffer = held.to_owned();
discard(&strip_markers(emit))
}
}
/// `str.find(sub, start)`.
fn find_from(text: &str, sub: &str, start: usize) -> Option<usize> {
text[start..].find(sub).map(|i| start + i)
}
fn strip_markers(text: &str) -> String {
[RESPONSE_OPEN, RESPONSE_CLOSE, MESSAGE_CLOSE]
.iter()
.fold(text.to_owned(), |text, marker| text.replace(marker, ""))
}
/// `_next_channel_idx`: the first `response` or `tools` channel at or after `start`.
fn next_channel_idx(text: &str, start: usize) -> Option<usize> {
[RESPONSE_OPEN, TOOLS_OPEN]
.iter()
.filter_map(|token| find_from(text, token, start))
.min()
}
/// `_skipped_think_channel`: a `response` close before any think close or
/// channel proves the text was the response channel.
fn skipped_think_channel(
text: &str,
start: usize,
think_close: Option<usize>,
next_channel: Option<usize>,
) -> bool {
find_from(text, RESPONSE_CLOSE, start).is_some_and(|close| {
[think_close, next_channel]
.iter()
.all(|boundary| boundary.is_none_or(|b| close < b))
})
}
/// `_is_skipped_think_answer`.
fn is_skipped_think_answer(text: &str) -> bool {
[THINK_OPEN, THINK_CLOSE, "<|open|>think", "<|close|>think"]
.iter()
.all(|token| !text.contains(token))
&& (text.contains(RESPONSE_CLOSE) || text.contains(MESSAGE_CLOSE))
}
/// `_clean_content`: unwrap the response channel; a `tools` channel stays raw.
fn clean_content(text: &str) -> String {
match text.find(TOOLS_OPEN) {
Some(tools) => strip_response_wrappers(&text[..tools]) + &text[tools..],
None => strip_response_wrappers(text),
}
}
@@ -3,15 +3,18 @@
mod deepseek_v4;
mod glm45;
mod kimi_k3;
pub(crate) use self::kimi_k3::KimiK3Detector;
use super::ThinkConfig;
/// The base-detector tokens a `--reasoning-parser` name selects.
/// The detector a `--reasoning-parser` name selects.
pub(crate) fn think_config(reasoning_parser: &str) -> Option<ThinkConfig> {
match reasoning_parser {
"deepseek-v4" => Some(deepseek_v4::THINK),
"deepseek-v41" => Some(deepseek_v4::THINK_V41),
"glm45" => Some(glm45::THINK),
"deepseek-v4" => Some(ThinkConfig::Tags(deepseek_v4::THINK)),
"deepseek-v41" => Some(ThinkConfig::Tags(deepseek_v4::THINK_V41)),
"glm45" => Some(ThinkConfig::Tags(glm45::THINK)),
"kimi_k3" => Some(ThinkConfig::KimiK3),
_ => None,
}
}
@@ -0,0 +1,199 @@
//! Kimi-K3 (`KimiK3Detector` in `function_call/kimik3_detector.py`, markers in
//! `kimik3_format.py`), with `FunctionCallParser`'s wrapping.
//!
//! Calls are numbered in output order and kept whatever their tool name. Typed
//! argument values go through `json.loads` and `json.dumps` as in Python
//! (`python_json_reencode`), except lone surrogates. `_ATTR_RE`'s `\w` is the
//! regex crate's, which differs from Python's only outside ASCII.
use std::sync::LazyLock;
use regex::Regex;
use serde_json::Value;
use crate::kimi_k3_format::{
CALL_CLOSE, CALL_OPEN, MESSAGE_CLOSE, RESPONSE_CLOSE, RESPONSE_OPEN, TOOLS_CLOSE, TOOLS_OPEN,
partial_suffix_len, strip_partial_marker_suffix, strip_response_wrappers,
};
use crate::py::{python_dict, python_json, python_json_reencode};
use crate::tool_call::{Parsed, ToolCallItem, ToolDetector};
const SEP: &str = "<|sep|>";
const ARGUMENT_OPEN: &str = "<|open|>argument";
const ARGUMENT_CLOSE: &str = "<|close|>argument<|sep|>";
/// `_ATTR_RE`.
static ATTR_RE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r#"(\w+)="([^"]*)""#).expect("valid attribute regex"));
#[derive(Default)]
pub(super) struct KimiK3Detector {
buffer: String,
/// `_sent_normal_idx`, a byte offset into `buffer`.
sent_normal_idx: usize,
/// `current_tool_id + 1`: calls already streamed.
streamed_calls: usize,
}
impl KimiK3Detector {
/// `detect_and_parse`, as `parse_non_stream` sees it once a tools channel is
/// present: the text after the channel is dropped.
fn detect_and_parse(&self, text: &str, open_idx: usize) -> Parsed {
let before = strip_response_wrappers(&text[..open_idx]);
let section = &text[open_idx + TOOLS_OPEN.len()..];
let section = section.find(TOOLS_CLOSE).map_or(section, |c| &section[..c]);
let calls = parse_calls(section)
.into_iter()
.enumerate()
.map(|(i, (name, parameters))| ToolCallItem {
tool_index: i as i64,
name,
parameters,
});
(before, calls.collect())
}
/// `parse_streaming_increment`: the section is not cut at `TOOLS_CLOSE`.
fn parse_streaming_increment(&mut self, text: &str) -> Parsed {
self.buffer.push_str(text);
let Some(open_idx) = self.buffer.find(TOOLS_OPEN) else {
return (self.emit_normal_text(None), Vec::new());
};
let normal = self.emit_normal_text(Some(open_idx));
let parsed = parse_calls(&self.buffer[open_idx + TOOLS_OPEN.len()..]);
let mut calls = Vec::new();
for (name, parameters) in parsed.into_iter().skip(self.streamed_calls) {
calls.push(ToolCallItem {
tool_index: self.streamed_calls as i64,
name,
parameters,
});
self.streamed_calls += 1;
}
(normal, calls)
}
/// `finish`: an open tools channel drops what is left; otherwise the held-back
/// tail is released, minus a cut marker.
fn finish(&mut self) -> Parsed {
if self.buffer.contains(TOOLS_OPEN) {
return Default::default();
}
let pending = self.emit_normal_text(Some(self.buffer.len()));
(strip_partial_marker_suffix(&pending).to_owned(), Vec::new())
}
/// `_emit_normal_text`: unsent text up to `limit`, or up to a possible marker
/// prefix at the end, minus the response/message wrappers.
fn emit_normal_text(&mut self, limit: Option<usize>) -> String {
let limit = limit.unwrap_or_else(|| {
let markers = [TOOLS_OPEN, RESPONSE_OPEN, RESPONSE_CLOSE, MESSAGE_CLOSE];
self.buffer.len() - partial_suffix_len(&self.buffer, &markers)
});
if limit <= self.sent_normal_idx {
return String::new();
}
let mut pending = self.buffer[self.sent_normal_idx..limit].to_owned();
for marker in [RESPONSE_OPEN, RESPONSE_CLOSE, MESSAGE_CLOSE] {
pending = pending.replace(marker, "");
}
self.sent_normal_idx = limit;
pending
}
}
impl ToolDetector for KimiK3Detector {
fn has_tool_call(&self, text: &str) -> bool {
text.contains(TOOLS_OPEN)
}
/// `FunctionCallParser.parse_non_stream`: without a tools channel the text
/// comes back untouched, wrappers included.
fn parse_non_stream(&self, text: &str) -> Parsed {
match text.find(TOOLS_OPEN) {
Some(open_idx) => self.detect_and_parse(text, open_idx),
None => (text.to_owned(), Vec::new()),
}
}
/// `parse_stream_chunk`, plus `parse_stream_end` when `flush`.
fn parse_stream(&mut self, text: &str, flush: bool) -> Parsed {
let (mut normal, mut calls) = self.parse_streaming_increment(text);
if flush {
let (end_normal, end_calls) = self.finish();
normal.push_str(&end_normal);
calls.extend(end_calls);
}
(normal, calls)
}
}
/// `_parse_calls`: `(name, arguments JSON)` of each complete call with a tool name.
fn parse_calls(section: &str) -> Vec<(String, String)> {
blocks(section, CALL_OPEN, CALL_CLOSE)
.into_iter()
.filter_map(|(attrs, body)| decode_call(attrs, body))
.collect()
}
/// `_decode_call`: `type="string"` values stay raw text, the rest are JSON with
/// the raw text as fallback.
fn decode_call(attrs: &str, body: &str) -> Option<(String, String)> {
let name = attr(attrs, "tool").unwrap_or_default();
if name.is_empty() {
return None;
}
let arguments =
blocks(body, ARGUMENT_OPEN, ARGUMENT_CLOSE)
.into_iter()
.map(|(arg_attrs, raw)| {
let value = match attr(arg_attrs, "type").as_deref().unwrap_or("string") {
"string" => None,
_ => python_json_reencode(raw),
};
let value = value.unwrap_or_else(|| python_json(&Value::String(raw.into()), false));
(attr(arg_attrs, "key").unwrap_or_default(), value)
});
Some((name, python_dict(arguments)))
}
/// `(attrs, body)` of each `_CALL_RE` / `_ARG_RE` match: `open`, whitespace,
/// attributes up to the first `<|sep|>`, then a body up to the first `close`.
fn blocks<'a>(text: &'a str, open: &str, close: &str) -> Vec<(&'a str, &'a str)> {
let mut found = Vec::new();
let mut pos = 0;
while let Some(start) = text[pos..].find(open).map(|i| pos + i + open.len()) {
let rest = &text[start..];
let attrs_start = rest.len() - rest.trim_start_matches(py_space).len();
if attrs_start == 0 {
// `\s+` needs one space; the next match starts past this marker.
pos = start;
continue;
}
// A missing `<|sep|>` or `close` fails every later start too.
let Some(sep) = rest[attrs_start..].find(SEP).map(|i| attrs_start + i) else {
break;
};
let body = &rest[sep + SEP.len()..];
let Some(end) = body.find(close) else {
break;
};
found.push((&rest[attrs_start..sep], &body[..end]));
pos = start + sep + SEP.len() + end + close.len();
}
found
}
/// `_parse_attrs(attrs).get(name)`: the last `name="..."`, unescaped.
fn attr(attrs: &str, name: &str) -> Option<String> {
ATTR_RE
.captures_iter(attrs)
.filter(|c| &c[1] == name)
.last()
.map(|c| c[2].replace("&quot;", "\"").replace("&amp;", "&"))
}
/// Python's `re` `\s` (`str.isspace`), which also matches U+001C..U+001F.
fn py_space(c: char) -> bool {
c.is_whitespace() || ('\u{1c}'..='\u{1f}').contains(&c)
}
@@ -2,6 +2,7 @@
//! differently. Every other name goes to Dynamo.
mod deepseek_v4;
mod kimi_k3;
use super::ToolDetector;
@@ -10,12 +11,19 @@ pub(crate) fn tool_detector(
tool_parser: &str,
tool_names: Vec<String>,
) -> Option<Box<dyn ToolDetector>> {
let detector = match tool_parser {
"deepseekv4" => deepseek_v4::DsmlDetector::new(deepseek_v4::DSML_TAGS, tool_names),
"deepseekv41" => deepseek_v4::DsmlDetector::new(deepseek_v4::DSML_TAGS_V41, tool_names),
Some(match tool_parser {
"deepseekv4" => Box::new(deepseek_v4::DsmlDetector::new(
deepseek_v4::DSML_TAGS,
tool_names,
)),
"deepseekv41" => Box::new(deepseek_v4::DsmlDetector::new(
deepseek_v4::DSML_TAGS_V41,
tool_names,
)),
// `KimiK3Detector` keeps calls to tools the request does not list.
"kimi_k3" => Box::new(kimi_k3::KimiK3Detector::default()),
_ => return None,
};
Some(Box::new(detector))
})
}
#[cfg(test)]
File diff suppressed because one or more lines are too long
+114
View File
@@ -0,0 +1,114 @@
{"model": "moonshotai/Kimi-K3", "revision": "f831ab66814297da540d832a5235f8e904f29d06", "config": {"model_type": "kimi_k3", "architectures": ["KimiK3ForConditionalGeneration"]}, "bos_token_id": 163584, "cases": [
{"name":"default","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"system","request":{"messages":[{"role":"system","content":"Be brief."},{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\"<|sep|>Be brief.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":109,"token_sha256":"7243aa729f662c0875b9cad80caaf2edd51ed65a333c91545d24409d7acdf8a7"},
{"name":"ignored_fields","request":{"messages":[{"role":"System","content":"sys","name":"policy","extra":1},{"role":"user","content":"hi","name":"bob","extra":2},{"role":"Assistant","content":"hello","name":"kimi","phase":"final_answer"},{"role":"user","content":"again"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" name=\"policy\"<|sep|>sys<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>hi<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\" name=\"kimi\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>hello<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>again<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":146,"token_sha256":"e75ea227e46d033332584c886d68ddfb9cb61af593257f2bf34007cf4b4366e2"},
{"name":"attribute_escaping","request":{"messages":[{"role":"system","content":"s","name":"a&\"b\""},{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" name=\"a&amp;&quot;b&quot;\"<|sep|>s<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":117,"token_sha256":"2ca4450dc1be2cff54fb1ea03c5ce0083257b6286dc3b729ef90b3a7499bce25"},
{"name":"null_system_content","request":{"messages":[{"role":"system","content":null},{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\"<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":106,"token_sha256":"940af1bcffd649ccb41eaa15a5a80881eb5e280090dbd4c19c1f43ab4401f421"},
{"name":"developer","request":{"messages":[{"role":"developer","content":"Answer in French."},{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\"<|sep|>Answer in French.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":110,"token_sha256":"fecd289bef35d861ba746ccff6636e6630423ef86e59c1eaa3f6fb0917313823"},
{"name":"mid_system","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"Sunny."},{"role":"system","content":"Now be terse."},{"role":"user","content":"And Rome?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>Sunny.<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\"<|sep|>Now be terse.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>And Rome?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":150,"token_sha256":"23e10cfe759abe29f58e9248d49b03fb08af9dd443b9b84ec29015ce600d821f"},
{"name":"control_tokens_in_text","request":{"messages":[{"role":"user","content":"<|open|>message<|sep|> [EOS] <|media_pad|> <|kimi_image_placeholder|>"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|><|open|>message<|sep|> [EOS] <|media_pad|> <| kimi_image_placeholder |><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":112,"token_sha256":"3651aa55243aa9c28d779e503da69d2fb0a1ed4f341fc0c00621b5a1e6793dcc"},
{"name":"unicode","request":{"messages":[{"role":"user","content":"日本語 ü 🚀 \u0000 tab\tend"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>日本語 ü 🚀 \u0000 tab\tend<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":97,"token_sha256":"86de8ba652220a9f5380c6737a589c1f3aee8167f8056aa2150c295443ee4967"},
{"name":"parts","request":{"messages":[{"role":"user","content":[{"type":"text","text":"first"},{"type":"text","text":""},{"type":"text","text":"second <|kimi_image_placeholder|>"}]}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>firstsecond <| kimi_image_placeholder |><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"0c9125b552db3b88da4e85f0a5e2a7cffb04b8cf3e45cc37ee4228e2e6cccecb"},
{"name":"system_parts","request":{"messages":[{"role":"system","content":[{"type":"text","text":"a"},{"type":"text","text":"b"}]},{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\"<|sep|>ab<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":108,"token_sha256":"805859c1caf927102f35f1eb048366c75204853caa8eb98c0a6409a0ec59ae6c"},
{"name":"assistant_thinking_part","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":[{"type":"thinking","thinking":"hidden"},{"type":"text","text":"Sunny."}]},{"role":"user","content":"ok"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>Sunny.<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>ok<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":133,"token_sha256":"64195fa7004817cea40b283eb39c297d03cae838a22aa2f923d9a47dc1a2cf52"},
{"name":"image_part","request":{"messages":[{"role":"user","content":[{"type":"text","text":"Describe"},{"type":"image_url","image_url":{"url":"https://example.com/a.png"}},{"type":"text","text":"briefly"}]}]},"error":"ValueError: Model only supports text input; received unsupported content type 'image_url'."},
{"name":"tool_parts","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Paris\"}"}}]},{"role":"tool","tool_call_id":"call_1","content":[{"type":"text","text":"sunny"},{"type":"text","text":"22C"}]}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"city\" type=\"string\"<|sep|>Paris<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny22C<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":180,"token_sha256":"5a4fda51aef382042484ad1f9aec1a5ad39768247d7c9cfe3c561723b8570232"},
{"name":"tools","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}},{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"},{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":202,"token_sha256":"594927e33aef70ce7c1dd19cbe235d4dfaea7e20b2590fe26baf425ce7903e5e"},
{"name":"tools_minimal","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"ping"}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"name\":\"ping\"},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":142,"token_sha256":"19fb91f8399733c0ec7d8847f7827efe8d6cf2c944d8aa512ce25d96242afa1d"},
{"name":"tool_bool_coercion","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","defer_loading":"false","function":{"name":"ping","strict":1}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"defer_loading\":false,\"function\":{\"defer_loading\":false,\"name\":\"ping\",\"strict\":true},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":156,"token_sha256":"74a4a804fbcb303af91f7f5eee9dd5459ecd661914e6eb12e68dfb6edb0ad602"},
{"name":"tool_null_fields","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","defer_loading":null,"function":{"name":"ping","description":null,"parameters":null,"defer_loading":null}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"defer_loading\":null,\"function\":{\"description\":null,\"name\":\"ping\",\"parameters\":null},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":155,"token_sha256":"5a2b50a08e42889e2191b985aec4d7c51100853e284888eb1560e60410351dbe"},
{"name":"tool_defer_loading","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","defer_loading":true,"function":{"name":"ping"}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"defer_loading\":true,\"function\":{\"defer_loading\":true,\"name\":\"ping\"},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":152,"token_sha256":"426a0f4e3b287abdf6cded7e7ca959c185db711af81f241c7137b4687767276d"},
{"name":"tool_extra_fields","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","x":1,"function":{"name":"ping","y":2}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"name\":\"ping\"},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":142,"token_sha256":"19fb91f8399733c0ec7d8847f7827efe8d6cf2c944d8aa512ce25d96242afa1d"},
{"name":"tool_schema_type_aliases","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"q","parameters":{"type":"object","properties":{"s":{"type":"varchar(255)"},"n":{"type":"Int32"},"f":{"type":["float","double","null"]},"l":{"type":"array","items":{"type":"uuid"}}}}}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"name\":\"q\",\"parameters\":{\"properties\":{\"f\":{\"type\":[\"number\",\"null\"]},\"l\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"},\"n\":{\"type\":\"integer\"},\"s\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":184,"token_sha256":"1f559636fada367cab7062fec34e3239f67938b76ec9dfe8026de05800119efc"},
{"name":"tools_empty","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"tool_choice_auto","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}],"tool_choice":"auto"},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":181,"token_sha256":"fd1239e35e784ea5c9d1ccb52714789a0683dfe6758bba4c4fcdbcb06b974ece"},
{"name":"tool_choice_none","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}],"tool_choice":"none"},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-choice\"<|sep|>The system is invoked with `tool_choice=none`.\nYou MUST NOT call any tools in the next message.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":219,"token_sha256":"d86fc2696959f1cb430f859d31ce3b56f5704c3297fb3504cd679566b0da0df3"},
{"name":"tool_choice_required","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}],"tool_choice":"required"},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-choice\"<|sep|>The system is invoked with `tool_choice=required`.\nYou MUST call tools in the next message.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":217,"token_sha256":"3a07c6c6c60c629a5c0d4b82828c8278fefa7037515663de856e79dd3210abcb"},
{"name":"tool_choice_named","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}],"tool_choice":{"type":"function","function":{"name":"get_weather"}}},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":181,"token_sha256":"fd1239e35e784ea5c9d1ccb52714789a0683dfe6758bba4c4fcdbcb06b974ece"},
{"name":"choice_without_tools","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tool_choice":"required"},"error":"ValueError: Tools cannot be empty if tool choice is set to required."},
{"name":"kwargs_tool_choice","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"tool_choice":"required"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-choice\"<|sep|>The system is invoked with `tool_choice=required`.\nYou MUST call tools in the next message.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":131,"token_sha256":"745f351547b87d1ff8bb61b3512d32da1414080189691cc6d95dc8544f134dd4"},
{"name":"kwargs_tool_choice_auto","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}],"tool_choice":"required","chat_template_kwargs":{"tool_choice":"auto"}},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":181,"token_sha256":"fd1239e35e784ea5c9d1ccb52714789a0683dfe6758bba4c4fcdbcb06b974ece"},
{"name":"system_tools","request":{"messages":[{"role":"system","content":"","tools":[{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-declare\"<|sep|>## New Tools Available\nThe system dynamically extends the toolset via lazy-loading.\nYou have access to all existing and extended tools.\nHere are the specs for the extended tools.\n\n```json\n[{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":174,"token_sha256":"c1ab5b4e19c37a1904aab78b4e7e0951218ac193ec6d13ee4ecc62e09f9540a2"},
{"name":"system_tools_no_content","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"system","tools":[{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},{"role":"user","content":"now search"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-declare\"<|sep|>## New Tools Available\nThe system dynamically extends the toolset via lazy-loading.\nYou have access to all existing and extended tools.\nHere are the specs for the extended tools.\n\n```json\n[{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>now search<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":187,"token_sha256":"69fd55d085fbdf2cde4a9ace9507e2d7aae127b01ad2a2c6cd1236cad29f35ef"},
{"name":"system_tools_with_content","request":{"messages":[{"role":"system","content":"dropped","tools":[{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-declare\"<|sep|>## New Tools Available\nThe system dynamically extends the toolset via lazy-loading.\nYou have access to all existing and extended tools.\nHere are the specs for the extended tools.\n\n```json\n[{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":174,"token_sha256":"c1ab5b4e19c37a1904aab78b4e7e0951218ac193ec6d13ee4ecc62e09f9540a2"},
{"name":"developer_tools","request":{"messages":[{"role":"developer","content":"policy","tools":[{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},{"role":"user","content":"What is the weather in Paris?"}],"tool_choice":"required"},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-declare\"<|sep|>## New Tools Available\nThe system dynamically extends the toolset via lazy-loading.\nYou have access to all existing and extended tools.\nHere are the specs for the extended tools.\n\n```json\n[{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-choice\"<|sep|>The system is invoked with `tool_choice=required`.\nYou MUST call tools in the next message.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":210,"token_sha256":"480ecf693e6b79a6836724f7291f6b72c25eb7ccf3964dfb1eb29eeb7f4bae64"},
{"name":"message_tool_schema_type_aliases","request":{"messages":[{"role":"system","content":"","tools":[{"type":"function","function":{"name":"q","parameters":{"type":"object","properties":{"s":{"type":"varchar(255)"},"n":{"type":"Int32"},"f":{"type":["float","double","null"]},"l":{"type":"array","items":{"type":"uuid"}}}}}}]},{"role":"developer","content":"policy","tools":[{"type":"function","function":{"name":"r","parameters":{"type":"object","properties":{"s":{"type":"varchar(255)"},"n":{"type":"Int32"},"f":{"type":["float","double","null"]},"l":{"type":"array","items":{"type":"uuid"}}}}}}]},{"role":"user","content":"What is the weather in Paris?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-declare\"<|sep|>## New Tools Available\nThe system dynamically extends the toolset via lazy-loading.\nYou have access to all existing and extended tools.\nHere are the specs for the extended tools.\n\n```json\n[{\"function\":{\"name\":\"q\",\"parameters\":{\"properties\":{\"f\":{\"type\":[\"number\",\"null\"]},\"l\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"},\"n\":{\"type\":\"integer\"},\"s\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-declare\"<|sep|>## New Tools Available\nThe system dynamically extends the toolset via lazy-loading.\nYou have access to all existing and extended tools.\nHere are the specs for the extended tools.\n\n```json\n[{\"function\":{\"name\":\"r\",\"parameters\":{\"properties\":{\"f\":{\"type\":[\"number\",\"null\"]},\"l\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"},\"n\":{\"type\":\"integer\"},\"s\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":315,"token_sha256":"d7708d49bcd6769e31a2f9061eb28d393ab7ba0cda70f29b6034da17764b885b"},
{"name":"message_tools_choice_none","request":{"messages":[{"role":"system","content":"","tools":[{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},{"role":"user","content":"What is the weather in Paris?"}],"tool_choice":"none"},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-declare\"<|sep|>## New Tools Available\nThe system dynamically extends the toolset via lazy-loading.\nYou have access to all existing and extended tools.\nHere are the specs for the extended tools.\n\n```json\n[{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-choice\"<|sep|>The system is invoked with `tool_choice=none`.\nYou MUST NOT call any tools in the next message.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":212,"token_sha256":"a2b1a3db5469d233914da060bf5117de653c67243ac46671549901172513a668"},
{"name":"message_tools_empty","request":{"messages":[{"role":"system","content":"sys","tools":[]},{"role":"user","content":"What is the weather in Paris?"}],"tool_choice":"required"},"error":"ValueError: Tools cannot be empty if tool choice is set to required."},
{"name":"request_and_message_tools","request":{"messages":[{"role":"system","content":"","tools":[{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-declare\"<|sep|>## New Tools Available\nThe system dynamically extends the toolset via lazy-loading.\nYou have access to all existing and extended tools.\nHere are the specs for the extended tools.\n\n```json\n[{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":260,"token_sha256":"f498399f767f0c36e26436f2db853cd61442e212dda714f36f22f1faf996949f"},
{"name":"assistant_message_tools","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"ok","tools":[{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},{"role":"user","content":"go"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>ok<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>go<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":131,"token_sha256":"6b37281fe30db81daad69665128758219eeeab85c03dd4126fe1794d9599107d"},
{"name":"tool_name_with_dot","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"web.search"}}]},"known_gap":"Dynamo validates K3 tool names; the processor leaves these to the engine","prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"name\":\"web.search\"},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":143,"token_sha256":"8d69146f5b28db4b0c5116840b59b3e8ed9986a001cfe85a66ae26d2d36be311"},
{"name":"duplicate_tool_names","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}},{"type":"function","function":{"name":"search","parameters":{"type":"object","properties":{}}}}]},"known_gap":"Dynamo rejects duplicate K3 tool names; the processor leaves these to the engine","prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"},{\"function\":{\"name\":\"search\",\"parameters\":{\"properties\":{},\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":174,"token_sha256":"09fb3e8725c94dc3a86a04bc0c15f6b128ff5ba844c4a1f37e63378643cd025b"},
{"name":"tool_without_type","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"function":{"name":"ping"}}]},"known_gap":"Dynamo requires type=function on K3 tools; the processor leaves these to the engine","prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"name\":\"ping\"}}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":138,"token_sha256":"7fd76537ca07cb159e8782aef25640084510eb8759ad78e526d7e5f8a9b58351"},
{"name":"tool_history","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"city\": \"Paris\", \"unit\": \"c\"}"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"city\" type=\"string\"<|sep|>Paris<|close|>argument<|sep|><|open|>argument key=\"unit\" type=\"string\"<|sep|>c<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":291,"token_sha256":"4a1a5da53645bea65ea82c4254389724707bfe9b5c671f5c88279e063431d31d"},
{"name":"nested_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"a\":{\"b\":[1,2,{\"c\":\"d\"}]},\"flag\":true,\"none\":null,\"s\":\"x\\\"y\",\"u\":\"é\"}"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"a\" type=\"object\"<|sep|>{\"b\": [1, 2, {\"c\": \"d\"}]}<|close|>argument<|sep|><|open|>argument key=\"flag\" type=\"boolean\"<|sep|>true<|close|>argument<|sep|><|open|>argument key=\"none\" type=\"null\"<|sep|>null<|close|>argument<|sep|><|open|>argument key=\"s\" type=\"string\"<|sep|>x\"y<|close|>argument<|sep|><|open|>argument key=\"u\" type=\"string\"<|sep|>é<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":266,"token_sha256":"83c3a5148d014d8806bc59e41c17be0620240602f650338eaae8c4e6d0931fb6"},
{"name":"dict_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":{"city":"Paris","n":2,"list":[1,"a"]}}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"city\" type=\"string\"<|sep|>Paris<|close|>argument<|sep|><|open|>argument key=\"n\" type=\"number\"<|sep|>2<|close|>argument<|sep|><|open|>argument key=\"list\" type=\"array\"<|sep|>[1, \"a\"]<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":225,"token_sha256":"f3d44c7448d6e40efc206ec988f343715ba16587d56fb50afca4ad738dd799c7"},
{"name":"empty_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":""}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":175,"token_sha256":"62e9d353401e4ef854aede8e1db0ebf572e7ff3006d0f63b0ee780eba355fb4a"},
{"name":"null_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":null}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":175,"token_sha256":"62e9d353401e4ef854aede8e1db0ebf572e7ff3006d0f63b0ee780eba355fb4a"},
{"name":"empty_object_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{}"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":175,"token_sha256":"62e9d353401e4ef854aede8e1db0ebf572e7ff3006d0f63b0ee780eba355fb4a"},
{"name":"placeholder_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"q\":\"<|kimi_image_placeholder|>\",\"l\":[\"<|kimi_image_placeholder|>\"]}"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"q\" type=\"string\"<|sep|><| kimi_image_placeholder |><|close|>argument<|sep|><|open|>argument key=\"l\" type=\"array\"<|sep|>[\"<| kimi_image_placeholder |>\"]<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":217,"token_sha256":"70022f37779e82ed98530484b26b968554d62212a5d4b5c83015604c5c5a5ddb"},
{"name":"invalid_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"not json"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"known_gap":"string arguments that are not a JSON object are left to the engine","prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>json type=\"object\"<|sep|>not json<|close|>json<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":187,"token_sha256":"db44f65545d0fa0129476a8ed63b4e23e90002331e8f63f56b3589872852a9a8"},
{"name":"array_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"[1, 2]"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"known_gap":"string arguments that are not a JSON object are left to the engine","prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>json type=\"object\"<|sep|>[1, 2]<|close|>json<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":191,"token_sha256":"cebb869e7183f32206c071f0d5ad08a9b0aa6dd86051d20ea56597ab4ea70fdd"},
{"name":"whitespace_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":" "}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"known_gap":"string arguments that are not a JSON object are left to the engine","prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>json type=\"object\"<|sep|> <|close|>json<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":186,"token_sha256":"fd7f476e333429b7e7259e16dbe918935c42c528958d08f37d4831680e2b00d1"},
{"name":"float_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"x\": 0.5, \"y\": 100.0, \"z\": 1e15, \"n\": -1.5, \"s\": 0.0001, \"o\": 0.0}"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"x\" type=\"number\"<|sep|>0.5<|close|>argument<|sep|><|open|>argument key=\"y\" type=\"number\"<|sep|>100.0<|close|>argument<|sep|><|open|>argument key=\"z\" type=\"number\"<|sep|>1000000000000000.0<|close|>argument<|sep|><|open|>argument key=\"n\" type=\"number\"<|sep|>-1.5<|close|>argument<|sep|><|open|>argument key=\"s\" type=\"number\"<|sep|>0.0001<|close|>argument<|sep|><|open|>argument key=\"o\" type=\"number\"<|sep|>0.0<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":284,"token_sha256":"4058ef859d5e1667d12791dcfb9450e2e799f1385fe0d0a1ee8c70611d5cdf51"},
{"name":"exponent_float_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"x\": 1e-06, \"w\": 1e16}"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"known_gap":"serde_json prints these floats unlike Python's json.dumps; left to the engine","prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"x\" type=\"number\"<|sep|>1e-06<|close|>argument<|sep|><|open|>argument key=\"w\" type=\"number\"<|sep|>1e+16<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":211,"token_sha256":"ba27fb927236660248b7beda39f5c63510954bab745725c52933431c10656cf6"},
{"name":"big_int_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"n\": 100000000000000000000}"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"known_gap":"serde_json reads this integer as a float; left to the engine","prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"n\" type=\"number\"<|sep|>1e+20<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":193,"token_sha256":"edded2832da8510b8afeca4708a5ecb6419d94ea481fe0465bb120c80584b093"},
{"name":"negative_zero_arguments","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"a\": -0, \"b\": -0.0}"}}]},{"role":"tool","tool_call_id":"call_1","content":"sunny"},{"role":"user","content":"Thanks"}]},"known_gap":"serde_json reads -0 as the float -0.0; left to the engine","prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"a\" type=\"number\"<|sep|>0<|close|>argument<|sep|><|open|>argument key=\"b\" type=\"number\"<|sep|>-0.0<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":208,"token_sha256":"b7149c411b6d76b5e4a3d49a4ef92226dd659b600d354c165deb4dccb4fb1017"},
{"name":"exponent_float_schema","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"f","parameters":{"type":"number","minimum":1e-05}}}]},"known_gap":"serde_json prints these floats unlike Python's json.dumps; left to the engine","prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"name\":\"f\",\"parameters\":{\"minimum\":1e-05,\"type\":\"number\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":156,"token_sha256":"3e0bb7ab538943998ce7e8606ef90072799a9462f831fc1ba24006ccaad9d1c8"},
{"name":"tool_results_reordered","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"a","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Paris\"}"}},{"id":"b","type":"function","function":{"name":"search","arguments":"{}"}}]},{"role":"tool","tool_call_id":"b","content":"results"},{"role":"tool","tool_call_id":"a","content":"sunny"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"city\" type=\"string\"<|sep|>Paris<|close|>argument<|sep|><|close|>call<|sep|><|open|>call tool=\"search\" index=\"2\"<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"search\" index=\"2\"<|sep|>results<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":212,"token_sha256":"15fdf4d0e93058764d8610489c01ccfb703a3aa1c77fa8a75107024b153862c2"},
{"name":"tool_result_positional_names","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"Checking.","tool_calls":[{"id":null,"type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Paris\"}"}},{"id":null,"type":"function","function":{"name":"search","arguments":"{}"}}]},{"role":"tool","content":"sunny"},{"role":"tool","name":"explicit","content":"results"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>Checking.<|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"city\" type=\"string\"<|sep|>Paris<|close|>argument<|sep|><|close|>call<|sep|><|open|>call tool=\"search\" index=\"2\"<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"explicit\" index=\"2\"<|sep|>results<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":214,"token_sha256":"ad9041df2cb3ee7d3708b2790ef3b60e151ce82546b12831f05f38181420d9a2"},
{"name":"tool_unresolvable","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"tool","content":"orphan"}]},"error":"ValueError: Kimi K3 tool messages need a resolvable tool name: carry `tool`/`name`, or match a preceding assistant tool_call by order."},
{"name":"assistant_name_and_calls","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"Let me check.","tool_calls":[{"id":"c","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"P\"}"}}],"name":"kimi"},{"role":"tool","tool_call_id":"c","content":"ok"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\" name=\"kimi\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>Let me check.<|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"city\" type=\"string\"<|sep|>P<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>ok<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":186,"token_sha256":"1623224ac882aa510de94d690831b3a9010d6f5f377a7fbbb4006c12eeca0b38"},
{"name":"reasoning_history","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"Sunny.","reasoning_content":"Check <|kimi_image_placeholder|>."},{"role":"user","content":"Rome?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>Check <| kimi_image_placeholder |>.<|close|>think<|sep|><|open|>response<|sep|>Sunny.<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Rome?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":142,"token_sha256":"03cd52ec20a5014abb90061410300fd1f0de9b67384352b3b2cf952cdc1792e4"},
{"name":"reasoning_history_no_thinking","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"Sunny.","reasoning_content":"Check."},{"role":"user","content":"Rome?"}],"chat_template_kwargs":{"thinking":false}},"prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>Sunny.<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Rome?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>","token_count":62,"token_sha256":"3306bbc3a2b9ad53b4ee39ab90d73d0f957ebba25ab0ba9d33cb420bb3e625e4"},
{"name":"reasoning_empty","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"Sunny.","reasoning_content":""},{"role":"user","content":"Rome?"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>Sunny.<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Rome?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":135,"token_sha256":"36a79965cac6af696f09d22b3cf526f9e106d12d77709fbc7ed33774c1e2aabd"},
{"name":"reasoning_whitespace","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"Sunny.","reasoning_content":" \n"},{"role":"user","content":"Rome?"}]},"known_gap":"Dynamo drops whitespace-only reasoning; the processor leaves it to the engine","prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|> \n<|close|>think<|sep|><|open|>response<|sep|>Sunny.<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Rome?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":136,"token_sha256":"318a5f8b2e5c4f9ee3634e7c5aef0dc4e35d5417c42aea907084352776033b67"},
{"name":"agentic_thinking","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null,"tool_calls":[{"id":"c1","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Paris\"}"}}],"reasoning_content":"Need weather."},{"role":"tool","tool_call_id":"c1","content":"sunny"},{"role":"assistant","content":"It is sunny.","reasoning_content":"Done."},{"role":"user","content":"Thanks"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}]},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>Need weather.<|close|>think<|sep|><|open|>response<|sep|><|close|>response<|sep|><|open|>tools<|sep|><|open|>call tool=\"get_weather\" index=\"1\"<|sep|><|open|>argument key=\"city\" type=\"string\"<|sep|>Paris<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"tool\" tool=\"get_weather\" index=\"1\"<|sep|>sunny<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>Done.<|close|>think<|sep|><|open|>response<|sep|>It is sunny.<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>Thanks<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":308,"token_sha256":"b53b00caf1cd0b205f01ea24d6ddf54596f4f36dbec6f6873baa06e5d0089d09"},
{"name":"thinking_false","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"thinking":false}},"prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>","token_count":28,"token_sha256":"27c0f76eeb397559f8f4ed800534faf0dc23e0cf5a51d7fb3b496fbdade003a8"},
{"name":"thinking_truthy_string","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"thinking":"false"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"thinking_zero","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"thinking":0}},"prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>","token_count":28,"token_sha256":"27c0f76eeb397559f8f4ed800534faf0dc23e0cf5a51d7fb3b496fbdade003a8"},
{"name":"thinking_null","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"thinking":null}},"prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>","token_count":28,"token_sha256":"27c0f76eeb397559f8f4ed800534faf0dc23e0cf5a51d7fb3b496fbdade003a8"},
{"name":"enable_thinking_only","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"enable_thinking":false}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"effort_none","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":"none"},"prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>","token_count":28,"token_sha256":"27c0f76eeb397559f8f4ed800534faf0dc23e0cf5a51d7fb3b496fbdade003a8"},
{"name":"effort_low","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":"low"},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=low`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":96,"token_sha256":"cd6840c93957a4cd43a1f5f9e21deb2ddce78195c3f75a2b52bf06c9ed5a614f"},
{"name":"effort_high","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":"high"},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=high`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":96,"token_sha256":"42fa21bff6c794f1d2ce0a4b580727320aeb1dcf64d5e384f1a97255887c8776"},
{"name":"effort_medium","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":"medium"},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"effort_float","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":0.5},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"effort_none_kwargs_thinking","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":"none","chat_template_kwargs":{"thinking":true}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"reasoning_object","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning":{"effort":"low"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=low`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":96,"token_sha256":"cd6840c93957a4cd43a1f5f9e21deb2ddce78195c3f75a2b52bf06c9ed5a614f"},
{"name":"reasoning_enabled_false","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning":{"enabled":false}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"reasoning_effort_none_object","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning":{"effort":"none"}},"prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>","token_count":28,"token_sha256":"27c0f76eeb397559f8f4ed800534faf0dc23e0cf5a51d7fb3b496fbdade003a8"},
{"name":"kwargs_effort","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"reasoning_effort":"low"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=low`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":96,"token_sha256":"cd6840c93957a4cd43a1f5f9e21deb2ddce78195c3f75a2b52bf06c9ed5a614f"},
{"name":"effort_precedence","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":"high","chat_template_kwargs":{"reasoning_effort":"low"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=low`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":96,"token_sha256":"cd6840c93957a4cd43a1f5f9e21deb2ddce78195c3f75a2b52bf06c9ed5a614f"},
{"name":"kwargs_effort_unsupported","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":"low","chat_template_kwargs":{"reasoning_effort":"medium"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"kwargs_thinking_effort","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"reasoning_effort":"low","chat_template_kwargs":{"thinking_effort":"high"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=high`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":96,"token_sha256":"42fa21bff6c794f1d2ce0a4b580727320aeb1dcf64d5e384f1a97255887c8776"},
{"name":"thinking_effort_invalid","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"thinking_effort":"medium"}},"error":"AssertionError: Unsupported thinking_effort='medium'; supported values are ['high', 'low', 'max']."},
{"name":"thinking_effort_invalid_no_thinking","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"thinking_effort":"medium","thinking":false}},"prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>","token_count":28,"token_sha256":"27c0f76eeb397559f8f4ed800534faf0dc23e0cf5a51d7fb3b496fbdade003a8"},
{"name":"thinking_effort_null","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"thinking_effort":null}},"known_gap":"Dynamo always renders the effort message; the processor leaves it to the engine","prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":28,"token_sha256":"dbcbbcf7234a21f3ce18ec7f8e88a59f9a76847a841b0c303d22345f4c4a9276"},
{"name":"kwargs_ignored","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"tokenize":false,"return_dict":true,"image_prompts":["x"],"other":1}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"final_assistant","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"It is"}]},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>It is<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":120,"token_sha256":"9aa08887e71e1e1fd532ae9d3b61ca1cb8fa2c3dbe1051a1c1c58fe593ba73f4"},
{"name":"continuation","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"It is"}],"continue_final_message":true},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>It is","token_count":97,"token_sha256":"c91b8d1373395de21b9c03ca40a31bef9a342d4737a2d6a66da9fd4c5e4b75b3"},
{"name":"continuation_coerced","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"It is"}],"continue_final_message":1},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>It is","token_count":97,"token_sha256":"c91b8d1373395de21b9c03ca40a31bef9a342d4737a2d6a66da9fd4c5e4b75b3"},
{"name":"continuation_special","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"<|close|>think<|sep|>Sun"}],"continue_final_message":true},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|>Sun","token_count":99,"token_sha256":"eb8b4cd0ea127dfb2afd897d5a09bceb576527eb53b703de4f95a68aba95b668"},
{"name":"continuation_bos","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"[BOS]It"}],"continue_final_message":true},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>[BOS]It","token_count":96,"token_sha256":"8da69b24cfc24da1c2147afde149e40d7fd4074b5eda3fdf10090043ade4dd61"},
{"name":"continuation_parts","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":[{"type":"text","text":"It is"}]}],"continue_final_message":true},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|><|close|>think<|sep|><|open|>response<|sep|>It is<|close|>response<|sep|><|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":120,"token_sha256":"9aa08887e71e1e1fd532ae9d3b61ca1cb8fa2c3dbe1051a1c1c58fe593ba73f4"},
{"name":"continuation_null","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":null}],"continue_final_message":true},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"continuation_only","request":{"messages":[{"role":"assistant","content":"Once"}],"continue_final_message":true},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>Once","token_count":78,"token_sha256":"e9a97f5201f8d321f66167e34507f36ce8002115d84e6fd193c1d2dd2066b3c9"},
{"name":"continuation_no_thinking","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"assistant","content":"It is"}],"continue_final_message":true,"chat_template_kwargs":{"thinking":false}},"prompt":"<|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>response<|sep|>It is","token_count":30,"token_sha256":"d9583f098407ce63f789d87dc49a2beae95b4afa161d2c26ba38813005b2206f"},
{"name":"json_object","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"response_format":{"type":"json_object"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_object`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":142,"token_sha256":"51dcf562443f60581915afa93bcb9021d6932d20bac2b10e57cae3638a10bac5"},
{"name":"json_schema","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"response_format":{"type":"json_schema","json_schema":{"name":"w","strict":true,"schema":{"type":"object","properties":{"z":{"type":"string"},"a":{"type":"number"}}}}}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_schema`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.\nThe JSON data must match the following schema:\n```json\n{\"properties\":{\"a\":{\"type\":\"number\"},\"z\":{\"type\":\"string\"}},\"type\":\"object\"}\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":175,"token_sha256":"74ada01eee21e955a9646bfeab7f5c91415264a6b89cc05fad7eb3c60853203f"},
{"name":"json_schema_without_schema","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"response_format":{"type":"json_schema","json_schema":{"name":"w"}}},"error":"ValueError: schema_ is required for json_schema response format request."},
{"name":"json_schema_legacy","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"response_format":{"type":"json_schema","schema":{"title":"T","type":"object","properties":{"b":{"type":"string"},"strict":{"default":true}}}}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_schema`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.\nThe JSON data must match the following schema:\n```json\n{\"properties\":{\"b\":{\"type\":\"string\"}},\"title\":\"T\",\"type\":\"object\"}\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":173,"token_sha256":"e9e877ef84c5852b92b3d0c9344381c6dac175f56bd4cfb00005e72c04876614"},
{"name":"response_format_text","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"response_format":{"type":"text"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":95,"token_sha256":"6dc54485fe04c9bcc40d8fc116c7c4a3d408a634918d08da211bbd794fade231"},
{"name":"kwargs_response_schema","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"response_format":{"type":"json_schema","json_schema":{"name":"w","schema":{"type":"object"}}},"chat_template_kwargs":{"response_schema":{"type":"string"}}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_schema`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.\nThe JSON data must match the following schema:\n```json\n{\"type\":\"string\"}\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":160,"token_sha256":"9b594a317db05f39b45d7c5cc52f0ca850d0d9ad2e4331854be8ff635f2ae37c"},
{"name":"kwargs_response_format","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"response_format":{"type":"text"},"chat_template_kwargs":{"response_format":{"type":"json_object"}}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_object`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":142,"token_sha256":"51dcf562443f60581915afa93bcb9021d6932d20bac2b10e57cae3638a10bac5"},
{"name":"kwargs_response_format_string","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"response_format":"json_object"}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_object`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":142,"token_sha256":"51dcf562443f60581915afa93bcb9021d6932d20bac2b10e57cae3638a10bac5"},
{"name":"kwargs_response_format_string_schema","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"response_format":"json_schema","response_schema":{"type":"string"}}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_schema`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.\nThe JSON data must match the following schema:\n```json\n{\"type\":\"string\"}\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":160,"token_sha256":"9b594a317db05f39b45d7c5cc52f0ca850d0d9ad2e4331854be8ff635f2ae37c"},
{"name":"json_schema_empty_with_legacy","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"response_format":{"type":"json_schema","json_schema":{},"schema":{"type":"object","title":"T"}}},"prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_schema`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.\nThe JSON data must match the following schema:\n```json\n{\"title\":\"T\",\"type\":\"object\"}\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":164,"token_sha256":"fb03f960ea57d540a6fb57bd42a46178fc05ff93786f5b2866cd3016723f0f64"},
{"name":"kwargs_truncation","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"truncation":true,"max_length":8}},"known_gap":"tokenizer arguments in chat_template_kwargs are left to the engine","prompt":"<|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":8,"token_sha256":"83fe4f71eea34824f0edc410d2eaa3f815f0a209943b9292c65f25ef1409799b"},
{"name":"kwargs_add_generation_prompt","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"add_generation_prompt":false}},"error":"TypeError: __main__.RecordingTokenizer.apply_chat_template() got multiple values for keyword argument 'add_generation_prompt'"},
{"name":"kwargs_messages","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"chat_template_kwargs":{"messages":[]}},"error":"TypeError: build_chat_segments() got multiple values for argument 'messages'"},
{"name":"response_format_with_tools","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Look up the weather.","parameters":{"type":"object","required":["city"],"properties":{"unit":{"enum":["c","f"]},"city":{"type":"string"}}}}}],"tool_choice":"required","response_format":{"type":"json_object"}},"prompt":"<|open|>message role=\"system\" type=\"tool-declare\"<|sep|># Tools\nHere are the available tools, described in JSONSchema.\n\n```json\n[{\"function\":{\"description\":\"Look up the weather.\",\"name\":\"get_weather\",\"parameters\":{\"properties\":{\"city\":{\"type\":\"string\"},\"unit\":{\"enum\":[\"c\",\"f\"]}},\"required\":[\"city\"],\"type\":\"object\"}},\"type\":\"function\"}]\n```<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"thinking-effort\"<|sep|>`thinking_effort` guides on how much to think in your thinking channel (not including the response channel), supported values include `low`, `medium`, `high`, and `max`.\nNow the system is invoked with `thinking_effort=max`.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"user\"<|sep|>What is the weather in Paris?<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"tool-choice\"<|sep|>The system is invoked with `tool_choice=required`.\nYou MUST call tools in the next message.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"system\" type=\"response-format\"<|sep|>The system is invoked with `response_format=json_object`.\nYour response must be raw JSON data without markdown code blocks (```json) or any additional formatting.<|close|>message<|sep|><|end_of_msg|><|open|>message role=\"assistant\"<|sep|><|open|>think<|sep|>","token_count":264,"token_sha256":"5768774c344f84d71099271b15a3668b65ba84adfe23f5611cf7e7ff7b49b4bf"},
{"name":"unknown_role","request":{"messages":[{"role":"function","name":"f","content":"x"},{"role":"user","content":"What is the weather in Paris?"}]},"error":"ValueError: Unknown message role 'function' at index 0."},
{"name":"latest_reminder_role","request":{"messages":[{"role":"user","content":"What is the weather in Paris?"},{"role":"latest_reminder","content":"x"}]},"error":"ValueError: Unknown message role 'latest_reminder' at index 1."}
]}
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -74,7 +74,7 @@ impl Checkpoint {
.ok()?;
let ids = |text: &str| self.tokenizer.encode(text).unwrap().token_ids().to_vec();
let bos = ids("").first().copied();
let mut out = ids(&prompt);
let mut out = ids(prompt.as_str());
let mut suffix = ids(&prefix);
if bos.is_some() && suffix.first().copied() == bos {
suffix.remove(0);
+18 -7
View File
@@ -8,9 +8,11 @@ use std::collections::HashMap;
use std::path::Path;
use serde_json::Value;
use sglang_processor::dynamo_renderer::RenderedPrompt;
use sglang_processor::dynamo_tokenizers::Tokenizer;
use sglang_processor::{
ChatFormatter, ChatFormatterOptions, load_tokenizer, resolve_model_file, select_chat_formatter,
ChatFormatter, ChatFormatterOptions, load_tokenizer, resolve_tokenizer_file,
select_chat_formatter,
};
use sha2::{Digest, Sha256};
@@ -83,8 +85,9 @@ fn check_case(
};
}
let (prompt, prefix) = rendered.map_err(|error| error.to_string())?;
if prompt.clone() + &prefix != case["prompt"] {
return Err(format!("prompt differs: {prompt}{prefix}"));
let text = prompt.as_str().to_owned() + &prefix;
if text != case["prompt"] {
return Err(format!("prompt differs: {text}"));
}
let Some(tokenizer) = tokenizer else {
return Ok(());
@@ -124,7 +127,7 @@ fn formatter(model: &str, fixture: &Value) -> ChatFormatter {
/// The pinned tokenizer when it is in the HF cache; otherwise say token ids are skipped.
fn cached_tokenizer(model: &str, revision: &str) -> Option<Tokenizer> {
let Some(file) = resolve_model_file(model, Some(revision), "tokenizer.json") else {
let Some(file) = resolve_tokenizer_file(model, Some(revision)) else {
eprintln!("{model}@{revision} is not in the HF cache: token ids not checked");
return None;
};
@@ -132,10 +135,18 @@ fn cached_tokenizer(model: &str, revision: &str) -> Option<Tokenizer> {
}
/// `serving_chat._append_assistant_prefix_to_prompt_ids`: the continuation is
/// encoded on its own, minus a leading BOS.
fn encode(tokenizer: &Tokenizer, prompt: &str, prefix: &str, bos: &Value) -> Vec<u32> {
/// encoded on its own, minus a leading BOS. Segmented prompts (Kimi K3) allow
/// special tokens only where the checkpoint's encoder does.
fn encode(tokenizer: &Tokenizer, prompt: &RenderedPrompt, prefix: &str, bos: &Value) -> Vec<u32> {
let ids = |text: &str| tokenizer.encode(text).unwrap().token_ids().to_vec();
let mut out = ids(prompt);
let mut out = match prompt.encode_segments() {
Some(segments) => tokenizer
.encode_segments(&segments)
.unwrap()
.token_ids()
.to_vec(),
None => ids(prompt.as_str()),
};
if !prefix.is_empty() {
let suffix = ids(prefix);
let skip = usize::from(suffix.first().is_some_and(|&id| *bos == id));
@@ -27,6 +27,7 @@ fn fixtures_match_sglang() {
let options = |stream_reasoning| ReasoningOptions {
force_reasoning: Some(case["force"] == true),
stream_reasoning,
force_nonempty_content: case["nonempty"] == true,
..Default::default()
};
let mut streaming =
@@ -309,6 +309,12 @@ def generate(path, model, engine_url):
name: value for name, value in defaults.items() if name not in SKIPPED_FIELDS
}
settings = fixture.get("settings", {})
# The encoder SGLang renders with; its OpenAI layer special-cases `kimi_k3`.
handler = lower(
{"endpoint": "chat", "request": BARE_REQUESTS["chat"]}, settings, model
)[0]
if handler.chat_encoding_spec is not None:
fixture["chat_model"]["chat_encoding_spec"] = handler.chat_encoding_spec
fixture["sampling_defaults"] = {
endpoint: lower({"endpoint": endpoint, "request": bare}, settings, model)[
2
@@ -51,6 +51,15 @@ class RecordingTokenizer:
self.texts.append(text)
return self.tokenizer.encode(text, **kwargs)
def apply_chat_template(self, *args, **kwargs):
# Kimi K3 tokenizes inside the template; record its text instead.
if kwargs.get("tokenize"):
text = self.tokenizer.apply_chat_template(
*args, **kwargs | {"tokenize": False}
)
self.texts.append(text)
return self.tokenizer.apply_chat_template(*args, **kwargs)
def __getattr__(self, name):
return getattr(self.tokenizer, name)
@@ -108,7 +117,11 @@ def generate(fixture_path):
# The server args `_validate_request` reads.
reset_context()
publish(ServerArgs(model_path=path), role="tokenizer")
tok = RecordingTokenizer(AutoTokenizer.from_pretrained(path, local_files_only=True))
tok = RecordingTokenizer(
AutoTokenizer.from_pretrained(
path, local_files_only=True, trust_remote_code=True
)
)
server = serving_chat(
fixture["model"],
path,
@@ -11,6 +11,7 @@ Run from rust/sglang-processor in a SGLang Python environment:
import json
import sys
from pathlib import Path
from types import SimpleNamespace
# Record this checkout's SGLang, not whichever one is installed.
sys.path.insert(0, str(Path(__file__).resolve().parents[4] / "python"))
@@ -22,11 +23,18 @@ from sglang.srt.parser.reasoning_parser import ReasoningParser
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures/reasoning_parity"
def split(parser, force, stream, chunks):
streaming = ReasoningParser(parser, stream_reasoning=stream, force_reasoning=force)
def split(parser, force, stream, chunks, nonempty=False):
# The request fields `ReasoningParser` reads.
kwargs = {"force_nonempty_content": True} if nonempty else {}
request = SimpleNamespace(chat_template_kwargs=kwargs, continue_final_message=False)
streaming = ReasoningParser(
parser, stream_reasoning=stream, force_reasoning=force, request=request
)
steps = [list(streaming.parse_stream_chunk(chunk)) for chunk in chunks]
steps.append(list(streaming.parse_stream_end()))
unary = ReasoningParser(parser, stream_reasoning=False, force_reasoning=force)
unary = ReasoningParser(
parser, stream_reasoning=False, force_reasoning=force, request=request
)
whole = unary.parse_non_stream("".join(chunks))
as_text = lambda pair: [part or "" for part in pair]
return {"steps": [as_text(step) for step in steps], "unary": as_text(whole)}
@@ -39,10 +47,15 @@ for path in sorted(FIXTURES.glob("*.json")):
"input": index,
"force": force,
"stream": stream,
**split(fixture["parser"], force, stream, case["chunks"]),
**({"nonempty": True} if nonempty else {}),
**split(fixture["parser"], force, stream, case["chunks"], nonempty),
}
for index, case in enumerate(fixture["inputs"])
for force in (True, False)
for stream in (True, False)
# `force_nonempty_content` cases only for the inputs that ask for them.
for nonempty in (
(False, True) if case.get("force_nonempty_content") else (False,)
)
]
path.write_text(dump(fixture))