# SGLang processor A thin wrapper over Dynamo's frontend crates (`dynamo-renderer`, `dynamo-tokenizers`, `dynamo-parsers`), shared by SGLang's Rust hosts. **Rule.** Each feature calls Dynamo directly. SGLang code exists only where Dynamo differs from SGLang's Python serving behavior, or lacks a model; that code names the Python source it mirrors. Hosts own request types, sampling, validation, transport and runtime. Hosts use the re-exported `dynamo_protocols`, `dynamo_renderer` and `dynamo_tokenizers` so their Dynamo versions match the processor's. The `render`, `tokenizer` and `parser` features (all default) gate each component and its Dynamo crate, so a host that only renders and tokenizes skips `dynamo-parsers`. ```text src/ model_files.rs find files in a model dir or the HF cache tokenizer/ encode prompts render/ chat request -> prompt models/ model-specific SGLang code legacy/ conversation.py templates parser/ engine output -> chat events openai/ OpenAI requests over /generate ``` ## model_files ```rust resolve_model_file(path, revision, filename) -> Option resolve_tokenizer_file(path, revision) -> Option ``` SGLang-only, because Dynamo has no file discovery. `path` is a file, a model directory, or an HF repo id that is read from the local HF cache without network access. Tokenizer discovery prefers `tiktoken.model` / `*.tiktoken` when `tokenizer_config.json` names a tiktoken class, then falls back to `tokenizer.json`. ## tokenizer ```rust load_tokenizer(path, revision, add_special_tokens) -> Result DynamoTokenizer::new(without_specials, with_specials) trait TextTokenizer { encode(text, add_special_tokens), encode_segments(segments, add_special_tokens) } ``` Pass-through. `load_tokenizer` calls `Tokenizer::from_file_with_options`, and `encode` / `encode_segments` call the Dynamo tokenizer. Dynamo fixes `add_special_tokens` at load time, so `DynamoTokenizer` holds one handle per setting and picks one per request. ## render ```rust select_chat_formatter(&ChatFormatterOptions) -> (Option, Option) load_chat_formatter(tokenizer_config, model_path, model_type, chat_template) -> Result ChatFormatter::render_prompt(&dyn OAIChatLikeRequest) -> Result ChatFormatter::render_request(serde_json::Value, &default_kwargs) -> Result<(RenderedPrompt, String), TemplateError> // DeepSeek-V4, V4.1, GLM-5.3, Kimi K3, Qwen3.8 DeepSeekV4Profile::from_checkpoint(profile_override, encoder_source) -> Result ChatFormatter::resolve_thinking(&mut kwargs, tools_enabled, named_tool_choice) -> Option ChatFormatter::stop_strs() -> Option> requested_effort(&body) -> Option<&Value> requested_thinking(&body) -> Option ``` `select_chat_formatter` (`selection.rs`) picks a formatter from `config.json`. A `ChatFormatterOptions::chat_template` override skips the native formatters. | Model | Formatter | |---|---| | DeepSeek V4 | `models/deepseek_v4.rs` on Dynamo's V4 encoder | | DeepSeek V4.1 | `models/deepseek_v41.rs` on Dynamo's V4.1 encoder | | GLM-5.3, GLM-5.3-Flash | `models/glm53.rs` on the checkpoint's Jinja template, picked by the template | | Kimi K3 | `models/kimi_k3.rs` on Dynamo's K3 formatter | | Qwen3.8 (Max, 27B, Flash-Next) | `models/qwen38.rs` on the checkpoint's Jinja template, picked by the template | | DeepSeek V3.2, Inkling, other Dynamo native models | Dynamo native formatter, with SGLang's thinking defaults | | Everything else | `load_chat_formatter` | `load_chat_formatter` (`loader.rs`) follows Python's `template_manager.py` order: a built-in name first; with no `--chat-template`, a legacy template inferred from the model path, then the tokenizer config's Jinja template; with a `--chat-template` file (`.jinja` or JSON), that file. On a missing template, the error string is returned so the host can report it per request. `render_prompt` is pass-through to `OAIPromptFormatter::render_prompt` for Jinja and native formatters. It keeps Dynamo's segments so Kimi K3 can encode with `encode_segments`. `render_request` takes the SGLang request body, because DeepSeek-V4 and V4.1 read `task`, `continue_final_message` and the `reasoning` object, which `OAIChatLikeRequest` lacks, and GLM-5.3 and Qwen3.8 build their template context from it as SGLang does. It returns the continuation prefix separately, since SGLang tokenizes it on its own. `default_kwargs` are the server's `--default-chat-template-kwargs`, which apply after the request's own values. On a DeepSeek formatter, `render_prompt` rebuilds the body from the trait; on Kimi K3 it keeps Dynamo's request semantics, which typed hosts rely on. `from_checkpoint` lets a host that fetches model files itself, such as sgl-router, build `ChatFormatter::DeepSeekV4` directly. `requested_effort` and `requested_thinking` (`reasoning.rs`) port `protocol.py`'s `reasoning` handling once, for the renderer and DeepSeek-V4 and V4.1 alike. The SGLang additions hook in as follows: | Code | Mirrors | Hook | |---|---|---| | `thinking.rs` | `parser/template_detection.py` | Reads the Jinja template's thinking toggle and its default; `resolve_thinking` writes that default into the kwargs before rendering. | | `legacy/` | `parser/conversation.py` | Replaces Dynamo: a native port of `Conversation.get_prompt()`, the built-in templates, and model-path inference. | | `models/deepseek_v4.rs` | `entrypoints/openai/serving_chat.py`, `encoding_dsv4.py`, `chat_encoding.py` | Replaces the OAI formatter. It normalises the request the way SGLang does (message dump, part flattening, tool field order, final assistant turn, `task`, thinking and effort, the checkpoint's effort profile), then calls `deepseek::v4::encode_messages_with_options`. | | `models/deepseek_v41.rs` | the same, `encoding_dsv41.py` | Replaces the OAI formatter. On top of the V4 dump it keeps text parts lists, sends only the request tool fields the client set, and maps effort to the encoder's 1-100 budget, then calls `deepseek::v41::encode_messages`. Developer turns and media are left to the engine. | | `models/glm53.rs` | `serving_chat.py` (`_process_messages`, `_apply_jinja_template`), `template_detection.py` | Picked by the template (`_is_glm53`). Builds the template context SGLang builds (message and tool dumps, parsed tool arguments, kwargs over the effort, server defaults), then renders it with Dynamo's Jinja formatter, text parts concatenated as the template does. Media, a `chat_template` kwarg, `-0` and U+001C..U+001F in assistant content are left to the engine. | | `models/kimi_k25.rs` | the Kimi K2.5 checkpoint's tool encoder | Adds `tools_ts_str` (TypeScript tool declarations) to the kwargs before the Jinja render. | | `models/kimi_k3.rs` | `serving_chat.py` (`_prepare_kimi_k3_messages`, `_encode_messages`), the checkpoint's `encoding_k3.py` | Replaces the OAI formatter's request handling. It prepares the messages, tools, thinking effort, `tool_choice` and `response_format` as SGLang does, then calls Dynamo's `KimiK3Formatter`. Shapes Dynamo renders differently are left to the engine. | | `models/qwen38.rs` | `serving_chat.py` (`_process_messages`, `_apply_jinja_template`), `jinja_template_utils.py` | Picked by the template. Builds the template context SGLang builds (message, tool and tool-call dumps, content as the template's content format takes it, kwargs over the effort, server defaults), then renders it with Dynamo's Jinja formatter. A later system turn fails as the template makes it fail, where Dynamo would demote it. Media, a `chat_template` kwarg, and text or numbers that Python's `trim` or `json.dumps` treat unlike Dynamo's are left to the engine. | `stop_strs` returns the legacy template's stop strings; Jinja and native formatters have none, as in Python. **Parity.** Each `tests/fixtures/parity/.json` holds requests that SGLang's Python serving path rendered and tokenized, written by `tests/scripts/generate_parity.py`. `tests/parity.rs` checks every fixture. To add a model, follow the [`processor-model-parity`](../../.claude/skills/processor-model-parity/SKILL.md) skill. ## parser ```rust ChatResponseProcessor::new(tool_parser, reasoning_parser, tools, tool_choice, uses_tool_call_structural_tag, parallel_tool_calls, choices) .with_reasoning_state(thinking) .process_stream(Stream) -> Stream split_reasoning(reasoning_parser, &options, text, token_ids) -> (reasoning, normal) ReasoningStreamSplitter::new(reasoning_parser, options).split(text, token_ids) tool_constraint(tool_parser, &dynamo_tool_choice(&tool_choice), &tools, parallel_tool_calls) -> Result, ProcessorError> tool_call_stream(tool_parser, tool_choice, tools, uses_tool_call_structural_tag, Stream) -> Stream parse_tool_calls(tool_parser, text, tools).await -> Result<(calls, normal), String> dynamo_tool_parser_name(sglang_name) -> &str ``` `mod.rs` holds the events and the stream processor, `reasoning.rs` the reasoning split, and `tools.rs` tool schemas, constraints and call deltas. `src/think/models/` and `src/tool_call/models/` list the parser names SGLang ports from Python because Dynamo's parsers split them differently; `src/think/` is the port of SGLang's `BaseReasoningFormatDetector` they configure and `src/tool_call/` of its `BaseFormatDetector`, both free of Dynamo so `openai` needs no `parser` feature. `tests/reasoning_parity.rs` and the test in `tool_call/models/mod.rs` check them against SGLang's `ReasoningParser` and `FunctionCallParser`. To port another model, add `think/models/.rs` or `tool_call/models/.rs` and its names in that `models/mod.rs`. A detector that overrides the base splitting, as Kimi-K3's does, is its own `ThinkConfig` variant. Tool detectors get the request's tools: Qwen3-Coder's types argument values by their schemas. Kimi-K3's `auto` tool-call structural tag is in `openai/kimi_k3_tag.rs`; `tests/fixtures/openai_parity/kimi-k3.json` checks it byte for byte. Pass-through for parsing: - Other reasoning parsers go through `ReasoningParserType::get_reasoning_parser_from_name` and `parse_reasoning_streaming_incremental`. - Other tool parsers go through `apply_tool_calling_jail`. Decoded output is wrapped as OpenAI stream chunks only to feed the parser, then unwrapped into `ChatEvent`. SGLang additions: - **Parser names** from SGLang are mapped onto Dynamo's before construction, mirroring `parser/reasoning_parser.py` and `function_call/function_call_parser.py`. - **Special tokens after a tool call** (qwen25 and glm47 terminators) are dropped. - **`parallel_tool_calls=false`** keeps only the first call. - **`tool_constraint`** turns `tool_choice` into the parser's structural tag, or a `json_schema` array of calls for `required` and named choices. ## openai ```rust lower_completion(body, &headers, &settings, tokenizer) -> Result<(generate_body, CompletionResponder), Unsupported> lower_chat(body, &headers, &settings, &chat_model, render, tokenizer) -> Result<(generate_body, ChatResponder), Unsupported> Responder::unary(status, body) / responder.stream_data(frame) -> OpenAI JSON or SSE events ``` SGLang-only: Python's OpenAI layer (`entrypoints/openai/`) over the engine's `/generate`, for hosts that send generation there. Lowering builds the `GenerateReqInput` SGLang's handler would, and the responder turns `/generate` output into the response SGLang's handler would return. Anything not reproduced exactly is `Unsupported`, and the host sends it to the engine's own route. `tests/openai_parity.rs` replays fixtures recorded by `tests/scripts/generate_openai_parity.py`, which runs SGLang's handler on the `/generate` output of a live engine, or on hand-written (`synthetic`) output. Chat splits reasoning with `think/`, and serves only the parser names `think/models/` ports. ## Host example ```rust let (formatter, error) = select_chat_formatter(&ChatFormatterOptions { tokenizer_path, model_path, ..Default::default() }); let formatter = formatter.ok_or(error)?; let thinking = formatter.resolve_thinking(&mut kwargs, tools_enabled, named_tool_choice); let prompt = formatter.render_prompt(&request)?; let ids = match prompt.encode_segments() { Some(segments) => tokenizer.encode_segments(&segments, false)?, None => tokenizer.encode(prompt.as_str(), false)?, }; let events = ChatResponseProcessor::new(/* ... */).with_reasoning_state(thinking).process_stream(decoded); ``` ```sh cargo test --manifest-path rust/Cargo.toml -p sglang-processor --locked ```