diff --git a/docs/cookbook/autoregressive/Kev/kev.mdx b/docs/cookbook/autoregressive/Kev/kev.mdx
new file mode 100644
index 0000000000..8972776385
--- /dev/null
+++ b/docs/cookbook/autoregressive/Kev/kev.mdx
@@ -0,0 +1,78 @@
+---
+title: Kev
+description: "Deploy the Kev decision models with SGLang. Kev returns calibrated probabilities for typed questions about a document, in sizes from 0.8B to 27B, each on one NVIDIA H200 in BF16."
+tag: NEW
+---
+
+## Deployment
+
+Every Kev size runs on one GPU in BF16 with TP=1 and needs no extra flag. SGLang reads the checkpoint's `head.pt` and configures the server.
+
+```bash Command
+sglang serve --model-path jaredpalmer/kev-27b --host 0.0.0.0 --port 30000
+```
+
+Replace the model path with any size in the table below. Each was validated on one H200.
+
+| Model | Hugging Face ID | Checkpoint | Backbone | GPU memory for weights |
+| --- | --- | --- | --- | --- |
+| Kev-27B | `jaredpalmer/kev-27b` | full weights | Qwen3.8-27B | 47.8 GB |
+| Kev-9B | `jaredpalmer/kev-9b` | LoRA adapter | Qwen3.5-9B-Base | 14.9 GB |
+| Kev-4B | `jaredpalmer/kev-4b` | LoRA adapter | Qwen3.5-4B-Base | 8.0 GB |
+| Kev-0.8B | `jaredpalmer/kev-0.8b` | LoRA adapter | Qwen3.5-0.8B-Base | 1.5 GB |
+
+For a LoRA adapter, SGLang loads the base revision that the adapter's `head.pt` pins and merges the adapter into its weights at startup, in FP32 with one rounding to BF16, as the reference code merges it. The base is downloaded on first use.
+
+## 1. Model introduction
+
+[Kev](https://huggingface.co/collections/jaredpalmer/kev) is a family of decision models published by Jared Palmer under the Apache-2.0 license, with [reference code](https://github.com/jaredpalmer/kev). Each model keeps a Qwen3.5 or Qwen3.8 backbone and replaces the LM head with a pointer head. Each question runs as its own prompt after the shared state. The head compares the final hidden state at each option's closing token with the one at the prompt's last token and returns one logit per option, with no text generation. A temperature stored in `head.pt` calibrates the probabilities.
+
+Pass a text string, JSON object, or JSON array as the state. `/v1/systemone` returns one answer per question: a `choice` over allowed options, a `noul` probability of true, or a `score` over ordered levels. JSON states are flattened into labeled text as the reference code does.
+
+## 2. Configuration tips
+
+- **Route.** Send requests to `/v1/systemone`, or set your System One client's base URL to the server. This includes TypeSafe SDK clients. See the [API reference](/docs/supported-models/decision_models#system-one-compatible-api) for request and response fields. Kev refuses `/v1/decisions`, chat, generation, and `/encode` requests.
+- **One prefill per question.** The model reads the state once for every question, as the reference does, so a request with many questions over a long state costs that many prefills. SGLang enables embedding mode and disables radix caching, chunked prefill, and CUDA graphs. Keep TP=1 and PP=1.
+- **Answers.** `probabilities` are the softmax of the head's logits divided by the calibrated temperature. `confidence` uses the System One adapter formulas, as Kev's own server does. SGLang does not round the values to four decimals. Answers omit `x_label_mass` because the head scores only the allowed options.
+- **Long states.** The reference server accepts states of up to 65,536 tokens. In SGLang each question's prompt must also fit one prefill, 16,384 tokens by default, so longer prompts are refused with an error that gives both lengths. Raising `--max-prefill-tokens` is not supported for this model yet, because the server does not reserve memory for longer unchunked prefills and can run out of memory under load.
+- **Inputs.** Requests are text only, so `images` and `chat_template_kwargs` are refused. A `score` question takes up to 10 levels and a `choice` question up to 255 options.
+- **Not supported.** Kev-27B v1, the `v1-lora` revision of `jaredpalmer/kev-27b`, is an adapter trained on a BF16 backbone that the reference applies unmerged, so SGLang refuses it and serves the current full weight release instead. Kev-0.5B is a prototype on a Qwen2.5 base and is refused too.
+
+## 3. Advanced usage
+
+### 3.1 TypeSafe SDK
+
+```bash Command
+pip install typesafe-sdk
+```
+
+
+
+```python Example
+from typesafe_sdk import Choice, Noul, TypeSafeClient
+
+client = TypeSafeClient(api_key="local", base_url="http://localhost:30000", model="kev-latest")
+response = client.system_one(
+ state="I was charged twice for order 1182. Please refund one of the charges.",
+ questions={
+ "team": Choice(
+ instructions="Which team should handle this?",
+ criteria={"billing": "Charges and refunds", "shipping": "Deliveries", "returns": "Exchanges"},
+ ),
+ "urgent": Noul(instructions="Does this need a reply today?"),
+ },
+)
+print(response.choices["team"].choice, response.nouls["urgent"].noul)
+```
+
+
+
+
+
+Captured with Kev-27B on one H200. Probabilities can vary with the model size, hardware, and numerical backend.
+
+```text Output
+billing 0.7591662905989526
+```
+
+
diff --git a/docs/cookbook/autoregressive/intro.mdx b/docs/cookbook/autoregressive/intro.mdx
index 3c5d4fa248..f772cd92b7 100644
--- a/docs/cookbook/autoregressive/intro.mdx
+++ b/docs/cookbook/autoregressive/intro.mdx
@@ -211,4 +211,9 @@ metatags:
href="/cookbook/autoregressive/Cloudflare/clef"
img="/cards/logos/cloudflare.png"
/>
+
diff --git a/docs/docs.json b/docs/docs.json
index 336a788a1b..630f69a770 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -1523,6 +1523,12 @@
"pages": [
"cookbook/autoregressive/Cloudflare/clef"
]
+ },
+ {
+ "group": "Kev",
+ "pages": [
+ "cookbook/autoregressive/Kev/kev"
+ ]
}
]
},
diff --git a/docs/docs/supported-models/decision_models.mdx b/docs/docs/supported-models/decision_models.mdx
index c9b5e392df..5e6ca4d664 100644
--- a/docs/docs/supported-models/decision_models.mdx
+++ b/docs/docs/supported-models/decision_models.mdx
@@ -159,7 +159,7 @@ The request fails with HTTP 400 for:
- a server launched with `--enable-mis` or `--dllm-algorithm`, or one that uses a built-in conversation template, whether named with `--chat-template`, loaded from a JSON template file, or inferred from the model path
- a model whose chat route uses a built-in encoder instead of a chat template, a tokenizer that does not encode the rendered chat text back to the same ids, or a model that is not a generation model
- a chat template that raises an error for the question message, or a server launched with `--skip-tokenizer-init`
-- a decision checkpoint such as [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B), or a joint schema checkpoint such as [Clef](/cookbook/autoregressive/Cloudflare/clef), which answer through `/v1/systemone`
+- a decision checkpoint such as [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B), or a joint schema or pointer head checkpoint such as [Clef](/cookbook/autoregressive/Cloudflare/clef) or [Kev](/cookbook/autoregressive/Kev/kev), which answer through `/v1/systemone`
Errors about one question name its id or its position in `questions`.
@@ -207,7 +207,7 @@ These models mix full and linear attention. Answer probabilities can move by up
## System One compatible API
-`POST /v1/systemone` serves the same decisions in the request and response shape of the System One API, with a `state`, a map of `noul`, `choice`, and `score` questions keyed by your ids, and one answer per id. Clients written for that API, including the official TypeSafe SDKs, get decisions by pointing their base URL at the server, with the exceptions listed below. For ordinary generation models, the route uses the same rendering, label checks, and scoring as `/v1/decisions`, and refuses the same servers, chat templates, and prompts, listed in [Errors](#errors). Its request fields are checked as described below. A decision checkpoint such as [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B) instead gets the prompt, answer codes, and calibrated temperature it was trained with. When its `decision_config.json` sets `attention_mode` to `noncausal_full_attention`, as [PPLX-Decider-v1.1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B) does, its full-attention layers read the whole prompt, and the server turns off the radix cache and chunked prefill, which would show those layers only part of it. A joint schema checkpoint such as [Clef](/cookbook/autoregressive/Cloudflare/clef) answers all questions from one prompt in one prefill, scored by its joint schema head, and does not use the chat template.
+`POST /v1/systemone` serves the same decisions in the request and response shape of the System One API, with a `state`, a map of `noul`, `choice`, and `score` questions keyed by your ids, and one answer per id. Clients written for that API, including the official TypeSafe SDKs, get decisions by pointing their base URL at the server, with the exceptions listed below. For ordinary generation models, the route uses the same rendering, label checks, and scoring as `/v1/decisions`, and refuses the same servers, chat templates, and prompts, listed in [Errors](#errors). Its request fields are checked as described below. A decision checkpoint such as [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B) instead gets the prompt, answer codes, and calibrated temperature it was trained with. When its `decision_config.json` sets `attention_mode` to `noncausal_full_attention`, as [PPLX-Decider-v1.1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B) does, its full-attention layers read the whole prompt, and the server turns off the radix cache and chunked prefill, which would show those layers only part of it. A joint schema checkpoint such as [Clef](/cookbook/autoregressive/Cloudflare/clef) answers all questions from one prompt in one prefill, scored by its joint schema head, and does not use the chat template. A pointer head checkpoint such as [Kev](/cookbook/autoregressive/Kev/kev) answers each question from its own prompt after the shared state, scored by its pointer head, and does not use the chat template either.
```bash Command
pip install typesafe-sdk
diff --git a/python/sglang/srt/arg_groups/model_hook.py b/python/sglang/srt/arg_groups/model_hook.py
index 67407f3992..5f79806258 100644
--- a/python/sglang/srt/arg_groups/model_hook.py
+++ b/python/sglang/srt/arg_groups/model_hook.py
@@ -984,12 +984,21 @@ def handle_model_capability_adjustments(server_args: Any):
# A Clef checkpoint's joint schema head reads the final hidden state of
# every prompt token in one pass, so each request is one complete prefill
# with no decode: embedding mode, without prefix reuse or chunking. This
- # also puts the FA backend on its raw K/V path, which skips the KV pool.
+ # also puts the FA backend on its raw K/V path, which skips the KV pool. A
+ # Kev checkpoint's pointer head reads two positions of each question's
+ # prompt and is served the same way.
if model_config.joint_head_config is not None:
+ name, head = "Clef", "joint schema head"
+ tp_reason = ", since the joint schema head reads full LM head rows"
+ elif model_config.pointer_head is not None:
+ name, head, tp_reason = "Kev", "pointer head", ""
+ else:
+ name = None
+ if name is not None:
if cfg.tp_size != 1 or cfg.pp_size != 1:
raise ValueError(
- "Clef checkpoints are served with --tp-size 1 and --pp-size 1, "
- "since the joint schema head reads full LM head rows"
+ f"{name} checkpoints are served with --tp-size 1 and --pp-size 1"
+ f"{tp_reason}"
)
for key, value in (
("is_embedding", True),
@@ -1008,7 +1017,7 @@ def handle_model_capability_adjustments(server_args: Any):
),
)
logger.info(
- "Clef joint schema head detected: serving /v1/systemone decisions in "
+ f"{name} {head} detected: serving /v1/systemone decisions in "
"embedding mode without radix cache, chunked prefill, or CUDA graphs."
)
diff --git a/python/sglang/srt/configs/model_config.py b/python/sglang/srt/configs/model_config.py
index 6006a5a902..0c32fd0476 100644
--- a/python/sglang/srt/configs/model_config.py
+++ b/python/sglang/srt/configs/model_config.py
@@ -35,6 +35,7 @@ from sglang.srt.layers.quantization import QUANTIZATION_METHODS
from sglang.srt.runtime_context import get_platform
from sglang.srt.server_args import ServerArgs
from sglang.srt.utils import is_hip, retry
+from sglang.srt.utils.hf_transformers.config import POINTER_ADAPTER_BASES
from sglang.srt.utils.hf_transformers_utils import (
get_config,
get_context_length,
@@ -308,6 +309,71 @@ def load_joint_head_config(model_path: str, revision: Optional[str]) -> Optional
return json.load(f)
+# The values the Kev reference reads for head.pt fields that older checkpoints
+# did not write.
+_POINTER_HEAD_DEFAULTS = {
+ "base_revision": None,
+ "head_dim": 256,
+ "option_isolation": False,
+ "special_embeddings": False,
+ "weights_dtype": "fp32",
+ "temperature": 1.0,
+}
+
+
+@lru_cache
+def load_pointer_head(model_path: str, revision: Optional[str]) -> Optional[dict]:
+ """head.pt of a Kev checkpoint, whose pointer head answers /v1/systemone
+ decisions from the backbone's hidden states."""
+ path = cached_file(
+ model_path,
+ "head.pt",
+ revision=revision,
+ _raise_exceptions_for_missing_entries=False,
+ _raise_exceptions_for_connection_errors=False,
+ )
+ if path is None:
+ return None
+ meta = {
+ **_POINTER_HEAD_DEFAULTS,
+ **torch.load(path, map_location="cpu", weights_only=True),
+ }
+ # Both change the prompt or the attention mask the head was trained with.
+ for name in ("option_isolation", "special_embeddings"):
+ if meta[name]:
+ raise ValueError(f"head.pt {name} is not supported")
+ return meta
+
+
+@lru_cache
+def load_pointer_adapter(model_path: str, revision: Optional[str]) -> Optional[dict]:
+ """head.pt of a Kev LoRA checkpoint with its merge scale. Such a checkpoint has
+ no model config and is served as the base backbone its head.pt pins, with the
+ adapter merged into the base weights at load."""
+ meta = load_pointer_head(model_path, revision)
+ if meta is None:
+ return None
+ path = cached_file(
+ model_path,
+ "adapter_config.json",
+ revision=revision,
+ _raise_exceptions_for_missing_entries=False,
+ _raise_exceptions_for_connection_errors=False,
+ )
+ if path is None:
+ return None
+ # Kev applies an adapter trained on a BF16 backbone unmerged, which a merge
+ # into the base weights does not reproduce.
+ if meta["weights_dtype"] != "fp32":
+ raise ValueError(
+ f"{model_path} is a Kev adapter trained on a {meta['weights_dtype']} "
+ "backbone, which SGLang does not serve. Serve its full weight release."
+ )
+ with open(path) as f:
+ adapter = json.load(f)
+ return {**meta, "lora_scale": adapter["lora_alpha"] / adapter["r"]}
+
+
def is_qwen3_5(config) -> bool:
return _hf_arch(config) in (
"Qwen3_5ForConditionalGeneration",
@@ -548,6 +614,13 @@ class ModelConfig:
**kwargs,
)
)
+ self.pointer_head = None
+ base = POINTER_ADAPTER_BASES.get((self.model_path, revision))
+ if base is not None:
+ # A Kev LoRA checkpoint, whose base weights load with its adapter merged.
+ self.pointer_head = load_pointer_adapter(self.model_path, revision)
+ self.model_path, self.revision = base
+ revision = self.revision
self.hf_text_config = get_hf_text_config(self.hf_config)
self.decision_config = None
if self.hf_config.architectures == ["Qwen3_5Model"]:
@@ -564,6 +637,11 @@ class ModelConfig:
self.joint_head_config = None
if self.hf_config.architectures == ["Qwen3_5ForConditionalGeneration"]:
self.joint_head_config = load_joint_head_config(self.model_path, revision)
+ if self.hf_config.architectures == ["Qwen3_5TextModel"]:
+ self.pointer_head = load_pointer_head(self.model_path, revision)
+ if self.pointer_head is not None:
+ # A Kev backbone without an LM head, served by the text-only class.
+ self.hf_config.architectures = ["Qwen3_5ForCausalLM"]
self.requires_mm_token_modalities = requires_mm_token_modalities(
self.hf_config.architectures, self.hf_text_config
)
diff --git a/python/sglang/srt/entrypoints/http_server.py b/python/sglang/srt/entrypoints/http_server.py
index 0abcc2924d..bf00f65649 100644
--- a/python/sglang/srt/entrypoints/http_server.py
+++ b/python/sglang/srt/entrypoints/http_server.py
@@ -70,7 +70,7 @@ from uvicorn.supervisors.multiprocess import Multiprocess
from sglang.srt.arg_groups.overrides import resolving_view
from sglang.srt.configs.embedding_model_spec import resolved_embedding_plan
-from sglang.srt.configs.model_config import load_joint_head_config
+from sglang.srt.configs.model_config import load_joint_head_config, load_pointer_head
from sglang.srt.constants import HEALTH_CHECK_RID_PREFIX
from sglang.srt.disaggregation.utils import FAKE_BOOTSTRAP_HOST, DisaggregationMode
from sglang.srt.entrypoints.anthropic.protocol import (
@@ -2406,13 +2406,18 @@ def _execute_server_warmup(server_args: ServerArgs):
).tolist()
json_data["sampling_params"]["max_new_tokens"] = 0
- if (
- not model_info["is_generation"]
- and model_info.get("architectures") == ["Qwen3_5ForConditionalGeneration"]
- and load_joint_head_config(get_model().model_path, get_model().revision)
- is not None
+ model = get_model()
+ if not model_info["is_generation"] and (
+ (
+ model_info.get("architectures") == ["Qwen3_5ForConditionalGeneration"]
+ and load_joint_head_config(model.model_path, model.revision) is not None
+ )
+ or (
+ model_info.get("architectures") == ["Qwen3_5ForCausalLM"]
+ and load_pointer_head(model.model_path, model.revision) is not None
+ )
):
- # A Clef checkpoint answers /v1/systemone decisions only.
+ # A Clef or Kev checkpoint answers /v1/systemone decisions only.
request_name = "/v1/systemone"
json_data = {
"model": "warmup",
diff --git a/python/sglang/srt/entrypoints/openai/serving_decisions.py b/python/sglang/srt/entrypoints/openai/serving_decisions.py
index edf7698786..db65d94130 100644
--- a/python/sglang/srt/entrypoints/openai/serving_decisions.py
+++ b/python/sglang/srt/entrypoints/openai/serving_decisions.py
@@ -84,6 +84,7 @@ class OpenAIServingDecisions(OpenAIServingBase):
self.prompt_text_is_lossy = chat_serving._prompt_text_round_trip_is_lossy
self.decision_config = self.tokenizer_manager.model_config.decision_config
self.joint_head_config = self.tokenizer_manager.model_config.joint_head_config
+ self.pointer_head = self.tokenizer_manager.model_config.pointer_head
tokenizer = self.tokenizer_manager.tokenizer
# Other tokenizers skip the shortcut in _encode_labels and check the full prompt.
self.added_tokens = (
diff --git a/python/sglang/srt/entrypoints/systemone/kev.py b/python/sglang/srt/entrypoints/systemone/kev.py
new file mode 100644
index 0000000000..23422a2d34
--- /dev/null
+++ b/python/sglang/srt/entrypoints/systemone/kev.py
@@ -0,0 +1,127 @@
+"""The prompts that a Kev checkpoint's pointer head reads.
+
+Kev answers /v1/systemone questions from the backbone's hidden states instead
+of from generated text. Each question is its own prompt, the shared state
+followed by the question and its options between reserved delimiter tokens,
+and the head scores each option at its closing token. This follows encode and
+to_record in the checkpoint's reference code, kev/model.py and kev/api.py.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any, List, Tuple
+
+from sglang.srt.entrypoints.systemone.protocol import (
+ SystemOneChoiceQuestion,
+ SystemOneNoulQuestion,
+ SystemOneQuestion,
+ SystemOneRequest,
+)
+from sglang.srt.layers.joint_schema_head import (
+ QUESTION_TYPES,
+ LayoutQuestion,
+ pack_decision_layout,
+)
+
+# Rarely used Qwen tokens that the checkpoint reads as state, question, option,
+# option end, and decide delimiters.
+_DELIMITERS = (
+ "<|fim_prefix|>",
+ "<|fim_middle|>",
+ "<|box_start|>",
+ "<|box_end|>",
+ "<|fim_suffix|>",
+)
+# Text that looks like a delimiter is rewritten, so callers cannot forge one.
+_SPECIAL = re.compile(r"<\|([A-Za-z0-9_]+)\|>")
+_ESCAPED = "<¦\\1¦>"
+# The reference server's limits: the state with its delimiter, and one prompt.
+MAX_STATE_TOKENS = 65536
+MAX_ROW_TOKENS = MAX_STATE_TOKENS + 8192
+
+
+def _render(value: Any, indent: int = 0) -> str:
+ """Flattens JSON into the labeled text the checkpoint was trained on."""
+ pad = " " * indent
+ if value is None:
+ return ""
+ if isinstance(value, (str, int, float, bool)):
+ return str(value)
+ if isinstance(value, list):
+ return "\n".join(f"{pad}- {_render(x, indent + 1).lstrip()}" for x in value)
+ return "\n".join(
+ (
+ f"{pad}{key}:\n{_render(x, indent + 1)}"
+ if isinstance(x, (dict, list))
+ else f"{pad}{key}: {_render(x)}"
+ )
+ for key, x in value.items()
+ )
+
+
+def _option(name: str, description: Any) -> str:
+ if description is None or description == "":
+ return name
+ return f"{name}: {_render(description)}"
+
+
+def kev_options(question: SystemOneQuestion) -> List[str]:
+ """Each option's text, in the order the head scores them."""
+ if isinstance(question, SystemOneNoulQuestion):
+ criteria = question.criteria
+ return [
+ _option("no", criteria.false if criteria else None),
+ _option("yes", criteria.true if criteria else None),
+ ]
+ if isinstance(question, SystemOneChoiceQuestion):
+ return [_option(name, detail) for name, detail in question.criteria.items()]
+ return [_render(level) for level in question.criteria]
+
+
+def encode_kev(
+ tokenizer: Any, request: SystemOneRequest, max_prompt_tokens: int
+) -> List[Tuple[List[int], List[int]]]:
+ """Each question's prompt ids and decision layout, in request order."""
+
+ def tokens(value: Any) -> List[int]:
+ text = _SPECIAL.sub(_ESCAPED, _render(value))
+ return tokenizer.encode(text, add_special_tokens=False)
+
+ state_id, question_id, option_id, close_id, decide_id = (
+ tokenizer.convert_tokens_to_ids(token) for token in _DELIMITERS
+ )
+ prefix = [state_id] + tokens(request.state)
+ if len(prefix) > MAX_STATE_TOKENS:
+ raise ValueError(
+ f"the state is {len(prefix)} tokens with its delimiter, over the "
+ f"{MAX_STATE_TOKENS} token limit of this checkpoint"
+ )
+ limit = min(MAX_ROW_TOKENS, max_prompt_tokens)
+ rows = []
+ for name, question in request.questions.items():
+ input_ids = prefix + [question_id] + tokens(question.instructions)
+ option_spans = []
+ for option in kev_options(question):
+ start = len(input_ids)
+ input_ids += [option_id] + tokens(option) + [close_id]
+ option_spans.append((start, len(input_ids)))
+ input_ids.append(decide_id)
+ if len(input_ids) > limit:
+ raise ValueError(
+ f"question {name!r} needs a {len(input_ids)} token prompt, but "
+ f"this server scores at most {limit} in one prefill"
+ )
+ decide = len(input_ids) - 1
+ layout = pack_decision_layout(
+ len(input_ids),
+ [
+ LayoutQuestion(
+ QUESTION_TYPES.index(question.type),
+ (decide, decide + 1),
+ tuple(option_spans),
+ )
+ ],
+ )
+ rows.append((input_ids, layout))
+ return rows
diff --git a/python/sglang/srt/entrypoints/systemone/serving.py b/python/sglang/srt/entrypoints/systemone/serving.py
index c966e3b670..029b10eac8 100644
--- a/python/sglang/srt/entrypoints/systemone/serving.py
+++ b/python/sglang/srt/entrypoints/systemone/serving.py
@@ -33,6 +33,7 @@ from sglang.srt.entrypoints.systemone.joint_schema import (
encode_joint_schema,
joint_schema_options,
)
+from sglang.srt.entrypoints.systemone.kev import encode_kev
from sglang.srt.entrypoints.systemone.protocol import (
SystemOneChoiceAnswer,
SystemOneChoiceQuestion,
@@ -68,7 +69,7 @@ _INLINE_HEADER_CHARS = 1 << 16
class SystemOneServing(OpenAIServingDecisions):
"""Answers System One questions with the rendering, label checks, and scoring of
- /v1/decisions, or, for a Clef checkpoint, with its joint schema head."""
+ /v1/decisions, or, for a Clef or Kev checkpoint, with its own head."""
route = "/v1/systemone"
@@ -76,7 +77,7 @@ class SystemOneServing(OpenAIServingDecisions):
return "systemone-"
def _validate_request(self, request: SystemOneRequest) -> Optional[str]:
- if self.joint_head_config is not None:
+ if self.joint_head_config is not None or self.pointer_head is not None:
return self._validate_joint_schema_request(request)
return (
self._validate_server(request.model)
@@ -87,7 +88,7 @@ class SystemOneServing(OpenAIServingDecisions):
def _validate_joint_schema_request(
self, request: SystemOneRequest
) -> Optional[str]:
- """A Clef checkpoint renders its own prompt, without the chat template."""
+ """A Clef or Kev checkpoint renders its own prompt, without the chat template."""
route = self.route
if self.tokenizer_manager.tokenizer is None:
return f"{route} requires the server tokenizer"
@@ -101,8 +102,9 @@ class SystemOneServing(OpenAIServingDecisions):
return f"{route} images require a model served with multimodal input"
if request.chat_template_kwargs:
return (
- f"{route} renders the prompt this checkpoint's joint schema head "
- "was trained with, which takes no chat_template_kwargs"
+ f"{route} renders the prompt this checkpoint's "
+ f"{'pointer' if self.pointer_head is not None else 'joint schema'} "
+ "head was trained with, which takes no chat_template_kwargs"
)
return None
@@ -117,6 +119,8 @@ class SystemOneServing(OpenAIServingDecisions):
views = [_view(question) for question in request.questions.values()]
if self.joint_head_config is not None:
return self._joint_schema_request(request, views), (request, views)
+ if self.pointer_head is not None:
+ return self._pointer_request(request, views), (request, views)
encode = (
self._encoded_systemone_questions
if self.decision_config is None
@@ -142,6 +146,71 @@ class SystemOneServing(OpenAIServingDecisions):
or None,
)
+ def _pointer_request(
+ self, request: SystemOneRequest, views: List[QuestionView]
+ ) -> EmbeddingReqInput:
+ """One prompt per question, which the pointer head scores in its own prefill."""
+ for question_id, view in zip(request.questions, views):
+ if view.kind == "score":
+ _check_legend(question_id, view)
+ tokenizer_manager = self.tokenizer_manager
+ rows = encode_kev(
+ tokenizer_manager.tokenizer,
+ request,
+ max_prompt_tokens=max_joint_prompt_tokens(
+ context_len=tokenizer_manager.context_len,
+ num_reserved_tokens=tokenizer_manager.num_reserved_tokens,
+ max_req_input_len=tokenizer_manager.max_req_input_len,
+ max_prefill_tokens=get_schedule().max_prefill_tokens,
+ ),
+ )
+ return EmbeddingReqInput(
+ input_ids=[input_ids for input_ids, _ in rows],
+ decision_layout=[layout for _, layout in rows],
+ )
+
+ async def _answer_pointer(
+ self,
+ adapted_request: EmbeddingReqInput,
+ request: SystemOneRequest,
+ views: List[QuestionView],
+ raw_request: Request,
+ ) -> ORJSONResponse:
+ rets = await self.tokenizer_manager.generate_request(
+ adapted_request, raw_request
+ ).__anext__()
+ answers = {}
+ for question_id, view, ret in zip(request.questions, views, rets):
+ logits = ret["embedding"]
+ if len(logits) != len(view.names):
+ raise RuntimeError("the pointer head did not score this request")
+ top = max(logits)
+ weights = [math.exp(logit - top) for logit in logits]
+ total = math.fsum(weights)
+ probabilities = [weight / total for weight in weights]
+ # The head scores no then yes, which the view names yes and no.
+ answers[question_id] = _answer(
+ view=view,
+ probabilities=(
+ probabilities[::-1] if view.kind == "yes_no" else probabilities
+ ),
+ mass=1.0,
+ question_id=question_id,
+ )
+ response = SystemOneResponse(
+ model=self.tokenizer_manager.served_model_name,
+ answers=answers,
+ usage=SystemOneUsage(
+ input_tokens=sum(ret["meta_info"]["prompt_tokens"] for ret in rets)
+ ),
+ )
+ # The head scores only the allowed options, so there is no label mass.
+ return ORJSONResponse(
+ content=response.model_dump(
+ exclude={"answers": {"__all__": {"x_label_mass"}}}
+ )
+ )
+
async def _encode_joint_schema_prompt(
self, adapted_request: EmbeddingReqInput, request: SystemOneRequest
) -> None:
@@ -343,6 +412,10 @@ class SystemOneServing(OpenAIServingDecisions):
return await self._answer_joint_schema(
adapted_request, request, views, raw_request
)
+ if self.pointer_head is not None:
+ return await self._answer_pointer(
+ adapted_request, request, views, raw_request
+ )
_, _, result = await self._score(
adapted_request=adapted_request,
raw_request=raw_request,
diff --git a/python/sglang/srt/layers/pointer_head.py b/python/sglang/srt/layers/pointer_head.py
new file mode 100644
index 0000000000..3d76b0cbbb
--- /dev/null
+++ b/python/sglang/srt/layers/pointer_head.py
@@ -0,0 +1,76 @@
+"""Pointer head of Kev decision checkpoints.
+
+A Kev checkpoint is a Qwen3.5 backbone saved without an LM head, with its head
+in ``head.pt``. Each question runs as its own prompt after the shared state,
+and the head scores every option by the final hidden state at the option's
+closing token against the one at the prompt's last token.
+"""
+
+from __future__ import annotations
+
+import math
+from typing import List
+
+import torch
+from torch import nn
+
+from sglang.srt.layers.joint_schema_head import parse_decision_layout
+from sglang.srt.layers.pooler import EmbeddingPoolerOutput
+
+
+class PointerPooler(nn.Module):
+ """Option logits per request, from a one-question decision layout whose
+ question span ends at the prompt's last token and whose option spans end
+ at the options' closing tokens."""
+
+ def __init__(self, hidden_size: int, meta: dict) -> None:
+ super().__init__()
+ head_dim = meta["head_dim"]
+ # The checkpoint trains and serves the head in FP32.
+ self.q = nn.Linear(hidden_size, head_dim, dtype=torch.float32)
+ self.k = nn.Linear(hidden_size, head_dim, dtype=torch.float32)
+ self.scale = 1 / math.sqrt(head_dim)
+ self.temperature = float(meta["temperature"])
+
+ def forward(
+ self, hidden_states: torch.Tensor, forward_batch
+ ) -> EmbeddingPoolerOutput:
+ lengths = forward_batch.extend_seq_lens_cpu
+ embeddings: List[torch.Tensor] = [
+ hidden_states.new_empty(0, dtype=torch.float32)
+ for _ in range(forward_batch.batch_size)
+ ]
+ if forward_batch.decision_layouts is None or lengths is None:
+ return EmbeddingPoolerOutput(embeddings=embeddings)
+ prefix_lengths = forward_batch.extend_prefix_lens_cpu or [0] * len(lengths)
+ scored, decide, options, owners = [], [], [], []
+ offset = 0
+ for i, (layout, prefix, length) in enumerate(
+ zip(forward_batch.decision_layouts, prefix_lengths, lengths)
+ ):
+ # A layout that no longer fits its prompt gets no scores, which the
+ # decision route reports as a server error.
+ questions = None
+ if layout is not None and prefix == 0:
+ try:
+ questions = parse_decision_layout(layout, length)
+ except ValueError:
+ pass
+ if questions is not None and len(questions) == 1:
+ question = questions[0]
+ owners += [len(scored)] * len(question.option_spans)
+ scored.append((i, len(question.option_spans)))
+ decide.append(offset + question.question_span[1] - 1)
+ options += [offset + end - 1 for _, end in question.option_spans]
+ offset += length
+ if not scored:
+ return EmbeddingPoolerOutput(embeddings=embeddings)
+ device = hidden_states.device
+ h_decide = hidden_states[torch.tensor(decide, device=device)].float()
+ h_options = hidden_states[torch.tensor(options, device=device)].float()
+ owner = torch.tensor(owners, device=device)
+ logits = (self.k(h_options) * self.q(h_decide)[owner]).sum(-1) * self.scale
+ logits = logits / self.temperature
+ for (i, _), values in zip(scored, logits.split([n for _, n in scored])):
+ embeddings[i] = values
+ return EmbeddingPoolerOutput(embeddings=embeddings)
diff --git a/python/sglang/srt/managers/tokenizer_manager.py b/python/sglang/srt/managers/tokenizer_manager.py
index 807eef50e4..34fe4cfa83 100644
--- a/python/sglang/srt/managers/tokenizer_manager.py
+++ b/python/sglang/srt/managers/tokenizer_manager.py
@@ -1452,8 +1452,11 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
def _validate_joint_schema_request(
self, obj: Union[GenerateReqInput, EmbeddingReqInput], input_ids: List[int]
) -> None:
- """A Clef checkpoint scores the spans of a decision layout, and nothing else."""
- if self.model_config.joint_head_config is None:
+ """A Clef or Kev checkpoint scores the spans of a decision layout, and nothing else."""
+ if (
+ self.model_config.joint_head_config is None
+ and self.model_config.pointer_head is None
+ ):
return
if isinstance(obj, GenerateReqInput) or obj.decision_layout is None:
# Health checks only need a response, which is an empty embedding.
diff --git a/python/sglang/srt/models/qwen3_5_text.py b/python/sglang/srt/models/qwen3_5_text.py
index 5ef794d2cd..2929314dc5 100644
--- a/python/sglang/srt/models/qwen3_5_text.py
+++ b/python/sglang/srt/models/qwen3_5_text.py
@@ -17,10 +17,14 @@ import logging
from typing import Iterable, Optional, Set, Tuple, Union
import torch
+from safetensors.torch import load_file
from torch import nn
+from transformers.utils import cached_file
+from sglang.srt.configs.model_config import load_pointer_adapter, load_pointer_head
from sglang.srt.eplb.expert_location import ModelConfigForExpertLocation
from sglang.srt.layers.logits_processor import LogitsProcessor
+from sglang.srt.layers.pointer_head import PointerPooler
from sglang.srt.layers.quantization.base_config import QuantizationConfig
from sglang.srt.layers.utils import PPMissingLayer
from sglang.srt.layers.vocab_parallel_embedding import ParallelLMHead
@@ -29,12 +33,16 @@ from sglang.srt.model_loader.weight_utils import default_weight_loader
from sglang.srt.models import qwen3_5
from sglang.srt.models.qwen2_moe import Qwen2MoeSparseMoeBlock
from sglang.srt.platforms import current_platform
-from sglang.srt.runtime_context import get_parallel
+from sglang.srt.runtime_context import get_model, get_parallel
from sglang.srt.utils import LazyValue, add_prefix
logger = logging.getLogger(__name__)
_MODEL_PREFIX = "model."
+# Where a Kev LoRA checkpoint's base keeps the text backbone, and where the
+# adapter keeps its factors.
+_BASE_TEXT_PREFIX = "model.language_model."
+_ADAPTER_PREFIX = "base_model.model."
class Qwen3_5ForCausalLM(nn.Module):
@@ -70,7 +78,19 @@ class Qwen3_5ForCausalLM(nn.Module):
prefix=add_prefix("model", prefix),
)
- if self.pp_group.is_last_rank:
+ self.pooler = None
+ model = get_model()
+ meta = (
+ load_pointer_head(model.model_path, model.revision)
+ if model.is_embedding
+ else None
+ )
+ if meta is not None:
+ # A Kev checkpoint, whose pointer head scores options from the final
+ # hidden states, and which has no LM head.
+ self.pooler = PointerPooler(config.hidden_size, meta)
+ self.lm_head = PPMissingLayer()
+ elif self.pp_group.is_last_rank:
if self.pp_group.world_size == 1 and config.tie_word_embeddings:
self.lm_head = self.model.embed_tokens
else:
@@ -165,6 +185,7 @@ class Qwen3_5ForCausalLM(nn.Module):
forward_batch: ForwardBatch,
input_embeds: Optional[torch.Tensor] = None,
pp_proxy_tensors: Optional[PPProxyTensors] = None,
+ get_embedding: bool = False,
**kwargs,
) -> Union[torch.Tensor, PPProxyTensors]:
if self.is_mrope_enabled:
@@ -185,19 +206,55 @@ class Qwen3_5ForCausalLM(nn.Module):
if isinstance(hidden_states, tuple):
hidden_states, aux_hidden_states = hidden_states
+ if get_embedding and self.pooler is not None:
+ return self.pooler(hidden_states, forward_batch)
return self.logits_processor(
input_ids, hidden_states, self.lm_head, forward_batch, aux_hidden_states
)
+ def _pointer_adapter(self) -> dict:
+ """LoRA factors of a Kev LoRA checkpoint by backbone module, none for full weights."""
+ model = get_model()
+ meta = load_pointer_adapter(model.model_path, model.revision)
+ if meta is None:
+ return {}
+ tensors = load_file(
+ cached_file(
+ model.model_path, "adapter_model.safetensors", revision=model.revision
+ )
+ )
+ return {
+ name[len(_ADAPTER_PREFIX) : -len(".lora_A.weight")]: (
+ a,
+ tensors[name.replace(".lora_A.", ".lora_B.")],
+ meta["lora_scale"],
+ )
+ for name, a in tensors.items()
+ if name.endswith(".lora_A.weight")
+ }
+
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]) -> Set[str]:
params_dict = dict(self.named_parameters())
loaded_params: Set[str] = set()
+ lora = self._pointer_adapter() if self.pooler is not None else {}
+
# Keep prefix stripping lazy: materializing mmap-backed checkpoint tensors
# across model processes can exceed host memory before GPU copies finish.
def body_weights():
for name, loaded_weight in weights:
- if name.startswith(_MODEL_PREFIX):
+ if self.pooler is not None:
+ # Kev saves the backbone under its own names, and the base of a
+ # Kev LoRA checkpoint under its text model, next to an LM head
+ # that Kev does not use.
+ name = name.removeprefix(_BASE_TEXT_PREFIX)
+ if name.startswith("lm_head."):
+ continue
+ factors = lora.pop(name.removesuffix(".weight"), None)
+ if factors is not None:
+ loaded_weight = _merge_lora(loaded_weight, *factors)
+ yield name, loaded_weight
+ elif name.startswith(_MODEL_PREFIX):
yield name[len(_MODEL_PREFIX) :], loaded_weight
elif name == "lm_head.weight":
if self.config.tie_word_embeddings:
@@ -212,9 +269,21 @@ class Qwen3_5ForCausalLM(nn.Module):
loaded_params.add("lm_head.weight")
body_loaded = self.model.load_weights(body_weights())
+ if lora:
+ raise ValueError(
+ f"The Kev adapter has factors for {len(lora)} modules that the "
+ f"base checkpoint does not have, such as {next(iter(lora))}"
+ )
loaded_params.update(f"{_MODEL_PREFIX}{n}" for n in body_loaded)
- if self.config.tie_word_embeddings and self.pp_group.is_last_rank:
+ if self.pooler is not None:
+ model = get_model()
+ meta = load_pointer_head(model.model_path, model.revision)
+ self.pooler.load_state_dict(meta["head"], strict=True)
+ loaded_params.update(
+ f"pooler.{n}" for n, _ in self.pooler.named_parameters()
+ )
+ elif self.config.tie_word_embeddings and self.pp_group.is_last_rank:
loaded_params.add("lm_head.weight")
return loaded_params
@@ -255,4 +324,12 @@ class Qwen3_5MoeForCausalLM(Qwen3_5ForCausalLM):
)
+def _merge_lora(
+ weight: torch.Tensor, a: torch.Tensor, b: torch.Tensor, scale: float
+) -> torch.Tensor:
+ """The weight plus its LoRA delta, added in FP32 and rounded once, as the Kev
+ reference merges an adapter."""
+ return (weight.float() + scale * (b.float() @ a.float())).to(weight.dtype)
+
+
EntryClass = [Qwen3_5MoeForCausalLM, Qwen3_5ForCausalLM]
diff --git a/python/sglang/srt/utils/hf_transformers/config.py b/python/sglang/srt/utils/hf_transformers/config.py
index 899dd065a5..e80bbdac3d 100644
--- a/python/sglang/srt/utils/hf_transformers/config.py
+++ b/python/sglang/srt/utils/hf_transformers/config.py
@@ -13,8 +13,9 @@
# ==============================================================================
"""Config loading utilities."""
+import copy
from pathlib import Path
-from typing import Optional
+from typing import Dict, Optional, Tuple
from transformers import PretrainedConfig
from transformers.models.auto.modeling_auto import MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
@@ -47,6 +48,10 @@ from .common import (
from .gguf_native import build_gguf_config, has_native_gguf_support
from .mistral_utils import is_mistral_model, load_mistral_config
+# The base and revision that get_config read for each Kev LoRA checkpoint, which
+# has no model config of its own, keyed by checkpoint path and revision.
+POINTER_ADAPTER_BASES: Dict[Tuple[str, Optional[str]], Tuple[str, str]] = {}
+
def _set_architectures(config, arch_name):
config.update({"architectures": [arch_name]})
@@ -370,9 +375,32 @@ def get_config(
model_config_parser = "mistral" if is_mistral_model(model) else "hf"
parser = get_model_config_parser(model_config_parser)
- config = parser.parse(
- model, trust_remote_code=trust_remote_code, revision=revision, **kwargs
- )
+ try:
+ config = parser.parse(
+ model, trust_remote_code=trust_remote_code, revision=revision, **kwargs
+ )
+ except (OSError, ValueError):
+ # A Kev LoRA checkpoint has no model config. It serves the text backbone
+ # of the base its head.pt pins.
+ from sglang.srt.configs.model_config import load_pointer_adapter
+
+ meta = load_pointer_adapter(model, revision)
+ if meta is None:
+ raise
+ base = parser.parse(
+ meta["base"],
+ trust_remote_code=trust_remote_code,
+ revision=meta["base_revision"],
+ **kwargs,
+ )
+ if base.model_type != "qwen3_5":
+ raise ValueError(
+ f"Kev checkpoints on a {base.model_type} base are not supported, "
+ "only on Qwen3.5 and Qwen3.8 bases"
+ )
+ config = copy.deepcopy(base.text_config)
+ _set_architectures(config, "Qwen3_5ForCausalLM")
+ POINTER_ADAPTER_BASES[(model, revision)] = (meta["base"], meta["base_revision"])
if model_override_args:
if isinstance(config, DeepseekV41Config):
diff --git a/test/registered/unit/configs/test_multimodal_piecewise_cuda_graph.py b/test/registered/unit/configs/test_multimodal_piecewise_cuda_graph.py
index b4ab8da4f5..aeaa82d570 100644
--- a/test/registered/unit/configs/test_multimodal_piecewise_cuda_graph.py
+++ b/test/registered/unit/configs/test_multimodal_piecewise_cuda_graph.py
@@ -121,6 +121,7 @@ class TestMultimodalPiecewiseCudaGraph(CustomTestCase):
args._model_config = SimpleNamespace(
is_embedding_gemma=True,
joint_head_config=None,
+ pointer_head=None,
is_multimodal=False,
context_len=2048,
hf_config=SimpleNamespace(architectures=["Gemma3TextModel"]),
@@ -158,6 +159,7 @@ class TestMultimodalPiecewiseCudaGraph(CustomTestCase):
is_embedding_gemma=False,
),
joint_head_config=None,
+ pointer_head=None,
is_multimodal=False,
hf_config=SimpleNamespace(architectures=["BertModel"]),
)
diff --git a/test/registered/unit/entrypoints/openai/test_serving_decisions.py b/test/registered/unit/entrypoints/openai/test_serving_decisions.py
index 03e927a657..40f973d2da 100644
--- a/test/registered/unit/entrypoints/openai/test_serving_decisions.py
+++ b/test/registered/unit/entrypoints/openai/test_serving_decisions.py
@@ -160,6 +160,7 @@ class ScoringManager(TokenizerManagerScoreMixin):
),
decision_config=None,
joint_head_config=None,
+ pointer_head=None,
)
pad, video = (
tokenizer.encode(token, add_special_tokens=False)[0]
diff --git a/test/registered/unit/entrypoints/systemone/test_joint_schema.py b/test/registered/unit/entrypoints/systemone/test_joint_schema.py
index 3d03aada1d..3603960bed 100644
--- a/test/registered/unit/entrypoints/systemone/test_joint_schema.py
+++ b/test/registered/unit/entrypoints/systemone/test_joint_schema.py
@@ -1,5 +1,6 @@
"""Unit tests for /v1/systemone on Clef checkpoints: the joint schema prompt and its
-decision layout, and answers built from the joint schema head's option logits."""
+decision layout, and answers built from the joint schema head's option logits. Kev
+checkpoints: the per-question prompts and the pointer head positions."""
import asyncio
import base64
@@ -12,6 +13,7 @@ from types import SimpleNamespace
from unittest import mock
import pybase64
+import torch
from PIL import Image
from transformers import AutoTokenizer
@@ -19,6 +21,7 @@ from sglang.srt.entrypoints import http_server
from sglang.srt.entrypoints.openai.protocol import DecisionRequest
from sglang.srt.entrypoints.openai.serving_decisions import OpenAIServingDecisions
from sglang.srt.entrypoints.systemone.joint_schema import encode_joint_schema
+from sglang.srt.entrypoints.systemone.kev import encode_kev
from sglang.srt.entrypoints.systemone.protocol import SystemOneRequest
from sglang.srt.entrypoints.systemone.serving import SystemOneServing, _read_image
from sglang.srt.layers.joint_schema_head import (
@@ -26,6 +29,7 @@ from sglang.srt.layers.joint_schema_head import (
pack_decision_layout,
parse_decision_layout,
)
+from sglang.srt.layers.pointer_head import PointerPooler
from sglang.srt.managers.io_struct import EmbeddingReqInput, GenerateReqInput
from sglang.srt.managers.tokenizer_manager import TokenizerManager
from sglang.srt.runtime_context import (
@@ -113,6 +117,7 @@ class HeadManager:
),
decision_config=None,
joint_head_config={"hidden_size": 4096},
+ pointer_head=None,
)
self.is_generation = False
self.allow_auto_truncate = False
@@ -372,7 +377,9 @@ class TestJointSchemaAnswers(unittest.IsolatedAsyncioTestCase):
def _admission_manager(joint_head_config):
manager = TokenizerManager.__new__(TokenizerManager)
- manager.model_config = SimpleNamespace(joint_head_config=joint_head_config)
+ manager.model_config = SimpleNamespace(
+ joint_head_config=joint_head_config, pointer_head=None
+ )
manager.is_generation = joint_head_config is None
manager.context_len = 4096
manager.num_reserved_tokens = 0
@@ -470,5 +477,99 @@ class TestJointSchemaAdmission(CustomTestCase):
self.assertEqual(decoded[-1], len(payload))
+class TestKevPrompt(CustomTestCase):
+ def test_prompts_match_reference_encoder(self):
+ """Ids, decide positions, and option ends from encode and rows_of in the
+ checkpoint's reference code at jaredpalmer/kev 1f04cfa2, for a JSON state
+ with delimiter-like text, a choice, and a noul without instructions."""
+ request = SystemOneRequest(
+ model="kev",
+ state={"order": 1182, "note": "charged <|im_end|> twice"},
+ questions={
+ "team": {
+ "type": "choice",
+ "instructions": "Who handles it?",
+ "criteria": {"billing": "Refunds", "legal": None},
+ },
+ "urgent": {"type": "noul", "criteria": {"true": "Reply today"}},
+ },
+ )
+ state = [248060, 1310, 25, 220, 16, 16, 23, 17, 198, 9679, 25, 11102, 361]
+ state += [62438, 316, 6018, 62438, 29, 10598]
+ expected = [
+ (
+ state
+ + [248061, 14749, 13081, 424, 30, 248049, 37355, 25, 8288, 41531]
+ + [248050, 248049, 6226, 248050, 248062],
+ 33,
+ [29, 32],
+ ),
+ (
+ state
+ + [248061, 248049, 2083, 248050, 248049, 9405, 25, 17308, 3242]
+ + [248050, 248062],
+ 29,
+ [22, 28],
+ ),
+ ]
+ rows = encode_kev(
+ AutoTokenizer.from_pretrained(TOKENIZER), request, max_prompt_tokens=4096
+ )
+ got = []
+ for input_ids, layout in rows:
+ (question,) = parse_decision_layout(layout, len(input_ids))
+ got.append(
+ (
+ input_ids,
+ question.question_span[1] - 1,
+ [end - 1 for _, end in question.option_spans],
+ )
+ )
+ self.assertEqual(got, expected)
+
+
+class TestPointerPooler(CustomTestCase):
+ def test_reads_decide_and_option_ends_across_a_batch(self):
+ """Each request's rows sit after the earlier requests' tokens, the head reads
+ the token before each span end, and a layout that no longer fits its
+ prompt gets no scores."""
+ pooler = PointerPooler(4, {"head_dim": 2, "temperature": 2.0})
+ with torch.no_grad():
+ for linear in (pooler.q, pooler.k):
+ linear.weight.copy_(torch.eye(2, 4))
+ linear.bias.zero_()
+ # Position i of the flat batch holds (i, 1, 0, 0), so a logit names the two
+ # positions it read.
+ hidden = torch.stack(
+ [torch.tensor([float(i), 1.0, 0.0, 0.0]) for i in range(14)]
+ ).bfloat16()
+
+ def layout(length, decide, option_spans):
+ return pack_decision_layout(
+ length, [LayoutQuestion(1, (decide, decide + 1), option_spans)]
+ )
+
+ batch = SimpleNamespace(
+ batch_size=3,
+ extend_seq_lens_cpu=[5, 4, 5],
+ extend_prefix_lens_cpu=[0, 0, 0],
+ decision_layouts=[
+ layout(5, 4, ((1, 3), (3, 4))),
+ layout(4, 3, ((0, 1), (1, 2), (2, 3))),
+ layout(6, 5, ((1, 2),)),
+ ],
+ )
+ out = pooler(hidden, batch).embeddings
+
+ def logits(options, decide):
+ return torch.tensor(
+ [(option * decide + 1) / math.sqrt(2) / 2.0 for option in options]
+ )
+
+ torch.testing.assert_close(out[0], logits([2, 3], 4))
+ torch.testing.assert_close(out[1], logits([5, 6, 7], 8))
+ self.assertEqual(out[2].numel(), 0)
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/test/registered/unit/managers/test_hidden_state_server_mode.py b/test/registered/unit/managers/test_hidden_state_server_mode.py
index 5811973e53..a8fca1e7e7 100644
--- a/test/registered/unit/managers/test_hidden_state_server_mode.py
+++ b/test/registered/unit/managers/test_hidden_state_server_mode.py
@@ -26,7 +26,9 @@ class TestHiddenStateServerMode(CustomTestCase):
manager.allow_auto_truncate = False
manager.validate_total_tokens = False
manager.is_generation = True
- manager.model_config = SimpleNamespace(joint_head_config=None)
+ manager.model_config = SimpleNamespace(
+ joint_head_config=None, pointer_head=None
+ )
manager.server_args = SimpleNamespace(enable_custom_logit_processor=False)
manager._validate_token_ids_logprob = Mock()
return manager
diff --git a/test/registered/unit/managers/test_sampling_mask_validation.py b/test/registered/unit/managers/test_sampling_mask_validation.py
index 5412b01423..abfd8ccbb8 100644
--- a/test/registered/unit/managers/test_sampling_mask_validation.py
+++ b/test/registered/unit/managers/test_sampling_mask_validation.py
@@ -27,7 +27,9 @@ class TestSamplingMaskValidation(CustomTestCase):
self.manager.allow_auto_truncate = False
self.manager.validate_total_tokens = False
self.manager.is_generation = True
- self.manager.model_config = SimpleNamespace(joint_head_config=None)
+ self.manager.model_config = SimpleNamespace(
+ joint_head_config=None, pointer_head=None
+ )
def _validate(
self,
diff --git a/test/registered/unit/utils/test_hf_transformers_loading.py b/test/registered/unit/utils/test_hf_transformers_loading.py
index 49019dc489..f5ede1871d 100644
--- a/test/registered/unit/utils/test_hf_transformers_loading.py
+++ b/test/registered/unit/utils/test_hf_transformers_loading.py
@@ -27,6 +27,7 @@ from sglang.srt.utils.hf_transformers import (
get_rope_config,
get_tokenizer,
)
+from sglang.srt.utils.hf_transformers.config import POINTER_ADAPTER_BASES
from sglang.srt.utils.patch_tokenizer import unpatch_tokenizer
from sglang.test.ci.ci_register import register_cpu_ci
@@ -236,6 +237,50 @@ class TestHFTransformersLoading(unittest.TestCase):
self.assertEqual(batch["input_ids"].tolist(), [[3, 3, 3, 3, 5]])
self.assertEqual(batch["attention_mask"].tolist(), [[1, 1, 1, 1, 1]])
+ def test_kev_adapter_reads_the_text_config_of_its_pinned_base(self):
+ """A Kev LoRA checkpoint has no config.json. Its config is the text config
+ of the base its head.pt pins, served by the text-only class, and the base
+ is recorded so its weights load. A base without the Qwen3.5 backbone, as
+ Kev-0.5B's Qwen2.5, is refused, and so is an adapter trained on a BF16
+ backbone, as Kev-27B v1, which Kev applies unmerged."""
+ Path(self.model_path, "config.json").write_text(
+ json.dumps(
+ {
+ "model_type": "qwen3_5",
+ "architectures": ["Qwen3_5ForConditionalGeneration"],
+ "text_config": {"model_type": "qwen3_5_text", "hidden_size": 32},
+ }
+ )
+ )
+
+ def adapter(base, weights_dtype="fp32"):
+ directory = tempfile.TemporaryDirectory()
+ self.addCleanup(directory.cleanup)
+ Path(directory.name, "adapter_config.json").write_text(
+ json.dumps({"r": 16, "lora_alpha": 32})
+ )
+ torch.save(
+ {"base": base, "base_revision": None, "weights_dtype": weights_dtype},
+ Path(directory.name, "head.pt"),
+ )
+ return directory.name
+
+ path = adapter(self.model_path)
+ config = get_config(path, trust_remote_code=False, local_files_only=True)
+ self.assertEqual(
+ (config.model_type, config.architectures, config.hidden_size),
+ ("qwen3_5_text", ["Qwen3_5ForCausalLM"], 32),
+ )
+ self.assertEqual(POINTER_ADAPTER_BASES[(path, None)], (self.model_path, None))
+
+ llama = tempfile.TemporaryDirectory()
+ self.addCleanup(llama.cleanup)
+ Path(llama.name, "config.json").write_text(json.dumps(self.text_config()))
+ with self.assertRaisesRegex(ValueError, "on a llama base"):
+ get_config(adapter(llama.name), trust_remote_code=False)
+ with self.assertRaisesRegex(ValueError, "trained on a bf16 backbone"):
+ get_config(adapter(self.model_path, "bf16"), trust_remote_code=False)
+
if __name__ == "__main__":
unittest.main()