Last of four; stacked on #38048. The record is the operator's input; the bags are what is in effect. A reader that takes the record and reads a field off it gets the input, which is the wrong one of the two whenever resolution decided something -- and the mistake is silent, because for most fields and most launches the two agree. Several of these files already read both ways, sometimes in the same expression: ```python get_tokenizer( get_serving().tokenizer_path, tokenizer_mode=server_args.tokenizer_mode, # the input, not the decision ... ) ``` Sixty-odd files convert. Record field reads in runtime code go from 199 to 11. Nine parameters that the conversion emptied are dropped along with the argument at every call site -- the dead-parameter ratchet is what names them. ### "Runs after its process publishes" is a per-entry-point claim Most converted reads sit in the serving and model-executor layers, which only exist after publication, or in the two subprocess entry points, which publish first thing. Three places are not like that, and they keep reading the record they were handed: - **`HttpServerEngineAdapter`** launches the server as a *child*. The parent resolves the record and never publishes, so the adapter's own reads -- the launch banner, the API key in its readiness loop, the TP width in `update_weights_from_tensor` -- are of `self.server_args`. A bag read here fails closed in a bare process, or answers for an unrelated engine in one that happens to have published. - **`serve_grpc`** reads its sidecar port before the integrated servicer builds the `Engine` that publishes. The comment above that line already said so and already bound `cfg = resolving_view(server_args)` for it; the sidecar port and the port it derives from read `cfg`. - **`initialize_dp_attention`** runs from callers whose publish is not guaranteed, so its one predicate stays on the resolution view. `ROLE_NAMESPACE_SETS["dp_controller"]` gains `observability` and `serving`, because the controller's metrics gate, tracing setup and worker-port broadcast now read those namespaces. Under `SGLANG_ROLE_NAMESPACES=enforce` that set is what the process may read, so a conversion that reaches a new namespace has to widen it in the same change. ## Three things worth a reviewer's attention **Eleven reads were `getattr(record, "field", default)`.** An AST scan for attribute access does not see those, so the census that said "43 readers" was counting the shape it could match rather than the thing it was after. `incremental_streaming_output` was read that way twice, and the transcription tests were the only reason it surfaced. **Not every record read is a bag read waiting to happen.** A multimodal processor's `base_gpu_id` is the instance's, not the process's: two engines in one process keep different ones, and `test_publishing_another_config_does_not_move_the_device` exists to say so. It stays on the record while `rl_on_policy_target` beside it moves. `RequestMetricsExporter` is the same shape -- it is handed the directory it writes to, and a test builds several with different ones. `configure_logger` is a third: 17 call sites, one of which passes an `argparse.Namespace`, so it is not a global-context reader at all. Those eleven remaining reads are the ones with a reason. **The fixtures move with the code.** Tests that hung config off a mock manager now publish a record, which is what the serving layer reads; where a test states a value it says so with `override_server_args` instead of assigning through the mock. `test_hisparse_unit` is the last of them: it stubbed a `server_args` onto a fake scheduler to say the decode radix cache was off, and the value it was standing in for is the published default, so the stub goes and the class publishes. ## Two things CI caught that a local sweep could not **`unittest.TestCase.enterContext` is Python 3.11+.** The converted fixtures used it at 18 sites; `requires-python` is `>=3.10` and CI runs 3.10, so every one of them raised `AttributeError` there while passing on a newer local interpreter. They call `enter_override(self, ...)` now -- a four-line helper in `sglang/test/test_utils.py` over the override's own `install()` / `restore()`. **A batched sweep cannot see a missing publish.** Three fixtures needed a published config and did not have one; each *passed* inside a shard where some other file had published, and failed when run alone. The affected cases are `test_serving_completions` (which set `incremental_streaming_output` on the mock manager's record, where nothing reads it now), `test_qwen3_vl_feature_materialization` (same shape for `mm_enable_dp_encoder`), and the two Qwen Rust tests -- whose fixture already carried the comment `# Non-auto: get_resolved_model_impl would choke on a SimpleNamespace` next to the `model_impl` it sets, which is exactly what happened once `get_mm_processor_cls` started reading that value from the bag. Its `publish` mirrors `model_impl` now, like the four fields it already mirrored. ## Verification A full registered-unit sweep (648 files) against this stack's merge-base: 19 failures on both sides, the same 19, none of them config. That sweep is what caught 23 failures the file-scoped runs missed -- and, later, that the narrower 139-file list did not even contain the files this change reaches. It is also what caught the `test_hisparse_unit` fixture above: the file passes inside a shard where something else published, and fails when it is run on its own, which is why every failing file is re-run alone before it is counted.
401 lines
15 KiB
Python
401 lines
15 KiB
Python
# Copyright 2023-2024 SGLang Team
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
# ==============================================================================
|
|
"""
|
|
Centralized template management for chat templates and completion templates.
|
|
|
|
This module provides a unified interface for managing both chat conversation templates
|
|
and code completion templates, eliminating global state and improving modularity.
|
|
"""
|
|
|
|
import json
|
|
import logging
|
|
import os
|
|
from typing import Dict, Optional
|
|
|
|
from sglang.srt.managers.tokenizer_manager import TokenizerManager
|
|
from sglang.srt.parser.code_completion_parser import (
|
|
CompletionTemplate,
|
|
FimPosition,
|
|
completion_template_exists,
|
|
register_completion_template,
|
|
set_completion_template,
|
|
)
|
|
from sglang.srt.parser.conversation import (
|
|
Conversation,
|
|
SeparatorStyle,
|
|
chat_template_exists,
|
|
get_conv_template_by_model_path,
|
|
register_conv_template,
|
|
)
|
|
from sglang.srt.parser.jinja_template_utils import (
|
|
detect_jinja_template_content_format,
|
|
jinja_template_may_reorder_tool_results,
|
|
)
|
|
from sglang.srt.parser.template_detection import (
|
|
REASONING_PARSER_RULES,
|
|
TOOL_CALL_PARSER_RULES,
|
|
ReasoningToggleConfig,
|
|
build_detection_context,
|
|
detect_reasoning_pattern,
|
|
match_rules,
|
|
)
|
|
from sglang.srt.runtime_context import get_serving
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
class TemplateManager:
|
|
"""
|
|
Centralized manager for chat and completion templates.
|
|
|
|
This class encapsulates all template-related state and operations,
|
|
eliminating the need for global variables and providing a clean
|
|
interface for template management.
|
|
"""
|
|
|
|
def __init__(self):
|
|
self._chat_template_name: Optional[str] = None
|
|
self._completion_template_name: Optional[str] = None
|
|
self._jinja_template_content_format: Optional[str] = "openai"
|
|
self._force_reasoning: bool = False
|
|
self._reasoning_config: Optional[ReasoningToggleConfig] = None
|
|
self._suggested_reasoning_parser: Optional[str] = None
|
|
self._suggested_tool_call_parser: Optional[str] = None
|
|
self._jinja_template_may_reorder_tool_results: bool = False
|
|
|
|
@property
|
|
def chat_template_name(self) -> Optional[str]:
|
|
"""Get the current chat template name."""
|
|
return self._chat_template_name
|
|
|
|
@property
|
|
def completion_template_name(self) -> Optional[str]:
|
|
"""Get the current completion template name."""
|
|
return self._completion_template_name
|
|
|
|
@property
|
|
def jinja_template_content_format(self) -> Optional[str]:
|
|
"""Get the detected template content format ('string' or 'openai' or None)."""
|
|
return self._jinja_template_content_format
|
|
|
|
@property
|
|
def force_reasoning(self) -> bool:
|
|
"""
|
|
Check if the current chat template enforces reasoning/thinking.
|
|
|
|
Returns:
|
|
True if the template contains reasoning patterns like <think> tags
|
|
"""
|
|
return self._force_reasoning
|
|
|
|
@property
|
|
def reasoning_config(self) -> Optional[ReasoningToggleConfig]:
|
|
"""Get the reasoning toggle config inferred from chat template."""
|
|
return self._reasoning_config
|
|
|
|
@property
|
|
def suggested_reasoning_parser(self) -> Optional[str]:
|
|
"""Get the auto-detected reasoning parser name, or None."""
|
|
return self._suggested_reasoning_parser
|
|
|
|
@property
|
|
def suggested_tool_call_parser(self) -> Optional[str]:
|
|
"""Get the auto-detected tool-call parser name, or None."""
|
|
return self._suggested_tool_call_parser
|
|
|
|
@property
|
|
def jinja_template_may_reorder_tool_results(self) -> bool:
|
|
return self._jinja_template_may_reorder_tool_results
|
|
|
|
def _run_template_detection(self, template, tokenizer) -> None:
|
|
"""Run reasoning pattern and parser detection on a template."""
|
|
self._jinja_template_may_reorder_tool_results = (
|
|
jinja_template_may_reorder_tool_results(template)
|
|
)
|
|
self._force_reasoning, self._reasoning_config = detect_reasoning_pattern(
|
|
template
|
|
)
|
|
# Build context once, reuse for both parser detections (avoids
|
|
# duplicate tokenizer.get_vocab() calls).
|
|
ctx = build_detection_context(
|
|
template, tokenizer, self._reasoning_config, self._force_reasoning
|
|
)
|
|
if ctx is None:
|
|
return
|
|
self._suggested_reasoning_parser = match_rules(
|
|
ctx, REASONING_PARSER_RULES, "reasoning parser"
|
|
)
|
|
self._suggested_tool_call_parser = match_rules(
|
|
ctx, TOOL_CALL_PARSER_RULES, "tool-call parser"
|
|
)
|
|
|
|
def load_chat_template(
|
|
self,
|
|
tokenizer_manager: TokenizerManager,
|
|
chat_template_arg: Optional[str],
|
|
model_path: str,
|
|
) -> None:
|
|
"""
|
|
Load a chat template from various sources.
|
|
|
|
Args:
|
|
tokenizer_manager: The tokenizer manager instance
|
|
chat_template_arg: Template name, file path, or None to auto-detect
|
|
model_path: Path to the model
|
|
"""
|
|
if chat_template_arg:
|
|
self._load_explicit_chat_template(tokenizer_manager, chat_template_arg)
|
|
else:
|
|
# Guess chat template from model path
|
|
self.guess_chat_template_from_model_path(model_path)
|
|
|
|
# If no pre-defined template was found, fallback to HuggingFace template
|
|
if self._chat_template_name is None:
|
|
# Try HuggingFace template first
|
|
hf_template = self._resolve_hf_chat_template(tokenizer_manager)
|
|
if hf_template:
|
|
# override the chat template
|
|
if tokenizer_manager.tokenizer:
|
|
tokenizer_manager.tokenizer.chat_template = hf_template
|
|
self._jinja_template_content_format = (
|
|
detect_jinja_template_content_format(hf_template)
|
|
)
|
|
logger.info(
|
|
f"Using default HuggingFace chat template with detected content format: {self._jinja_template_content_format}"
|
|
)
|
|
else:
|
|
# Default to string content format if no template was found
|
|
self._jinja_template_content_format = "string"
|
|
logger.info(
|
|
"No chat template found, defaulting to 'string' content format"
|
|
)
|
|
|
|
# Detect reasoning pattern and suggest parser from chat template
|
|
if tokenizer_manager.tokenizer:
|
|
template = tokenizer_manager.tokenizer.chat_template
|
|
self._run_template_detection(template, tokenizer_manager.tokenizer)
|
|
parts = []
|
|
if self._reasoning_config:
|
|
parts.append(f"reasoning_config={self._reasoning_config}")
|
|
if self._suggested_reasoning_parser:
|
|
parts.append(f"reasoning_parser={self._suggested_reasoning_parser}")
|
|
if self._suggested_tool_call_parser:
|
|
parts.append(f"tool_call_parser={self._suggested_tool_call_parser}")
|
|
if parts:
|
|
logger.info(f"Auto-detected template features: {', '.join(parts)}")
|
|
|
|
def _load_explicit_chat_template(
|
|
self, tokenizer_manager: TokenizerManager, chat_template_arg: str
|
|
) -> None:
|
|
"""Load explicitly specified chat template."""
|
|
logger.info(f"Loading chat template from argument: {chat_template_arg}")
|
|
|
|
if chat_template_exists(chat_template_arg):
|
|
self._chat_template_name = chat_template_arg
|
|
return
|
|
|
|
if not os.path.exists(chat_template_arg):
|
|
raise RuntimeError(
|
|
f"Chat template {chat_template_arg} is not a built-in template name "
|
|
"or a valid chat template file path."
|
|
)
|
|
|
|
if chat_template_arg.endswith(".jinja"):
|
|
self._load_jinja_template(tokenizer_manager, chat_template_arg)
|
|
else:
|
|
self._load_json_chat_template(chat_template_arg)
|
|
|
|
def guess_chat_template_from_model_path(self, model_path: str) -> None:
|
|
"""
|
|
Infer chat template name from model path.
|
|
|
|
Args:
|
|
model_path: Path to the model
|
|
"""
|
|
template_name = get_conv_template_by_model_path(model_path)
|
|
if template_name is not None:
|
|
logger.info(f"Inferred chat template from model path: {template_name}")
|
|
self._chat_template_name = template_name
|
|
|
|
def load_completion_template(self, completion_template_arg: str) -> None:
|
|
"""
|
|
Load completion template for code completion.
|
|
|
|
Args:
|
|
completion_template_arg: Template name or file path
|
|
"""
|
|
logger.info(f"Loading completion template: {completion_template_arg}")
|
|
|
|
if not completion_template_exists(completion_template_arg):
|
|
if not os.path.exists(completion_template_arg):
|
|
raise RuntimeError(
|
|
f"Completion template {completion_template_arg} is not a built-in template name "
|
|
"or a valid completion template file path."
|
|
)
|
|
|
|
self._load_json_completion_template(completion_template_arg)
|
|
else:
|
|
self._completion_template_name = completion_template_arg
|
|
|
|
set_completion_template(self._completion_template_name)
|
|
|
|
def initialize_templates(
|
|
self,
|
|
tokenizer_manager: TokenizerManager,
|
|
model_path: str,
|
|
chat_template: Optional[str] = None,
|
|
completion_template: Optional[str] = None,
|
|
) -> None:
|
|
"""
|
|
Initialize all templates based on provided configuration.
|
|
|
|
Args:
|
|
tokenizer_manager: The tokenizer manager instance
|
|
model_path: Path to the model
|
|
chat_template: Optional chat template name/path
|
|
completion_template: Optional completion template name/path
|
|
"""
|
|
# Load chat template
|
|
self.load_chat_template(tokenizer_manager, chat_template, model_path)
|
|
|
|
# Load completion template
|
|
if completion_template:
|
|
self.load_completion_template(completion_template)
|
|
|
|
def _load_jinja_template(
|
|
self, tokenizer_manager: TokenizerManager, template_path: str
|
|
) -> None:
|
|
"""Load a Jinja template file."""
|
|
with open(template_path, "r") as f:
|
|
chat_template = "".join(f.readlines()).strip("\n")
|
|
tokenizer_manager.tokenizer.chat_template = chat_template.replace("\\n", "\n")
|
|
self._chat_template_name = None
|
|
# Detect content format from the loaded template
|
|
self._jinja_template_content_format = detect_jinja_template_content_format(
|
|
chat_template
|
|
)
|
|
logger.info(
|
|
f"Detected user specified Jinja chat template with content format: {self._jinja_template_content_format}"
|
|
)
|
|
|
|
def _load_json_chat_template(self, template_path: str) -> None:
|
|
"""Load a JSON chat template file."""
|
|
assert template_path.endswith(".json"), (
|
|
"unrecognized format of chat template file"
|
|
)
|
|
|
|
with open(template_path, "r") as filep:
|
|
template = json.load(filep)
|
|
try:
|
|
sep_style = SeparatorStyle[template["sep_style"]]
|
|
except KeyError:
|
|
raise ValueError(
|
|
f"Unknown separator style: {template['sep_style']}"
|
|
) from None
|
|
|
|
register_conv_template(
|
|
Conversation(
|
|
name=template["name"],
|
|
system_template=template["system"] + "\n{system_message}",
|
|
system_message=template.get("system_message", ""),
|
|
roles=(template["user"], template["assistant"]),
|
|
sep_style=sep_style,
|
|
sep=template.get("sep", "\n"),
|
|
stop_str=template["stop_str"],
|
|
),
|
|
override=True,
|
|
)
|
|
self._chat_template_name = template["name"]
|
|
|
|
def _load_json_completion_template(self, template_path: str) -> None:
|
|
"""Load a JSON completion template file."""
|
|
assert template_path.endswith(".json"), (
|
|
"unrecognized format of completion template file"
|
|
)
|
|
|
|
with open(template_path, "r") as filep:
|
|
template = json.load(filep)
|
|
try:
|
|
fim_position = FimPosition[template["fim_position"]]
|
|
except KeyError:
|
|
raise ValueError(
|
|
f"Unknown fim position: {template['fim_position']}"
|
|
) from None
|
|
|
|
register_completion_template(
|
|
CompletionTemplate(
|
|
name=template["name"],
|
|
fim_begin_token=template["fim_begin_token"],
|
|
fim_middle_token=template["fim_middle_token"],
|
|
fim_end_token=template["fim_end_token"],
|
|
fim_position=fim_position,
|
|
),
|
|
override=True,
|
|
)
|
|
self._completion_template_name = template["name"]
|
|
|
|
def _resolve_hf_chat_template(
|
|
self, tokenizer_manager: TokenizerManager
|
|
) -> Optional[str]:
|
|
try:
|
|
# Try (mm-)processor first, then tokenizer
|
|
template = (
|
|
getattr(tokenizer_manager.processor, "chat_template", None)
|
|
if tokenizer_manager.processor
|
|
else None
|
|
) or (
|
|
getattr(tokenizer_manager.tokenizer, "chat_template", None)
|
|
if tokenizer_manager.tokenizer
|
|
else None
|
|
)
|
|
|
|
if template is None:
|
|
logger.warning("No HuggingFace chat template found")
|
|
return None
|
|
|
|
# Handle dict templates (multiple named templates)
|
|
if isinstance(template, dict):
|
|
return self._select_named_template(template, tokenizer_manager)
|
|
|
|
# Single string template
|
|
return template
|
|
|
|
except Exception as e:
|
|
logger.warning(f"Error getting chat template: {e}")
|
|
return None
|
|
|
|
def _select_named_template(
|
|
self, templates: Dict[str, str], tokenizer_manager: TokenizerManager
|
|
) -> str:
|
|
if not templates:
|
|
raise ValueError("Empty templates dict provided")
|
|
|
|
available_names = list(templates.keys())
|
|
logger.info(f"Multiple HuggingFace chat templates available: {available_names}")
|
|
|
|
# Use specified template if provided
|
|
if preferred_name := get_serving().hf_chat_template_name:
|
|
if preferred_name not in templates:
|
|
raise ValueError(
|
|
f"Specified template '{preferred_name}' not found. "
|
|
f"Available templates: {available_names}"
|
|
)
|
|
logger.info(f"Using specified chat template: '{preferred_name}'")
|
|
return templates[preferred_name]
|
|
|
|
# Fallback: Use first available template
|
|
first_name = available_names[0]
|
|
logger.info(f"Using first available template: '{first_name}'")
|
|
return templates[first_name]
|