[Feature] support pplx-decider-v1.1 and update the pplx-decider cookbooks (#42645)

This commit is contained in:
Richard Wang
2026-10-08 09:03:50 +08:00
committed by GitHub
parent 9101e895ea
commit 5cb7fb886c
15 changed files with 1498 additions and 137 deletions
@@ -0,0 +1,563 @@
---
title: PPLX-Decider-v1-27B
description: "Deploy PPLX-Decider-v1-27B with SGLang. Perplexity's decision model, fine-tuned from Qwen3.8-27B, returns calibrated probabilities for choice, yes or no, and score questions on /v1/systemone, on one NVIDIA H200 or GB300 in BF16."
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. PPLX-Decider support landed after v0.5.21, so until a release includes it, install a [nightly build](/docs/get-started/install#nightly-builds).
<Tabs>
<Tab title="Python (pip / uv)">
```bash Command
pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
```
Then run the **Python** output of the command panel below in that environment.
</Tab>
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:dev
```
The image is multi-arch, so the same tag runs on x86 H200 hosts and Arm GB300 hosts. For how to launch it, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
Pick your GPU to generate the launch command. The model runs on one GPU in BF16, and the decision support needs no flag: SGLang reads the checkpoint's `decision_config.json`, loads its decision readout, and answers on `/v1/systemone` with the prompt, answer codes, and calibrated temperature the model was trained with. The GB300 recipe adds `--disable-radix-cache` for speed, as explained in [Configuration Tips](#2-configuration-tips).
import { Deployment } from "/src/snippets/_deployment.jsx"
import { config } from "/src/snippets/configs/perplexity-ai/pplx-decider-v1-27b.jsx"
import { benchmarks } from "/src/snippets/configs/perplexity-ai/pplx-decider-v1-27b-benchmarks.jsx"
<Deployment config={config} benchmarks={benchmarks} />
<Note>
The H200 recipe is the original one, validated against the checkpoint's own reference implementation on one H200 when support landed ([#42183](https://github.com/sgl-project/sglang/pull/42183)), and it has no speed numbers yet. The GB300 recipe adds `--disable-radix-cache` (see [Configuration Tips](#2-configuration-tips)) and was measured end to end on one GB300 at SGLang commit `70f0b7351e`: four request shapes from 1 to 256 concurrent requests, Belebele and WinoGrande accuracy, and per-question parity with the reference implementation. [§3](#3-benchmarks) has the method and the full tables.
</Note>
## Playground
The Playground layers SGLang features on top of the recipe above. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end. The [Configuration Tips](#2-configuration-tips) say which overrides were measured on GB300 and what they did. Tensor parallelism times data-parallel replicas must fit the GPUs on one node.
import { Playground } from "/src/snippets/_playground.jsx"
<Playground config={config} />
## 1. Model Introduction
**PPLX-Decider-v1-27B** is a decision model from Perplexity, fine-tuned from [Qwen3.8-27B](/cookbook/autoregressive/Qwen/Qwen3.8-27B) under the Apache-2.0 license. It replaces the language model head with a readout over 255 answer codes and answers choice, yes or no (`noul`), and score questions about a text state and optional images with a probability for every option, without generating text. Each question is one prefill pass, so the cost of a decision is the cost of reading its prompt. Its successor, [PPLX-Decider-v1.1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B), adds noncausal full attention and more training data.
The backbone is the Qwen3.8-27B hybrid: 64 layers in which 48 Gated DeltaNet (linear attention) layers alternate with 16 gated full-attention layers, a 5120-wide hidden state, and the Qwen3.8 vision encoder, for 26.1B parameters (52 GB in BF16). The checkpoint ships a bare `Qwen3_5Model` backbone, the 255-row readout in `readout.safetensors`, and `decision_config.json` with the answer-code token ids and a calibrated temperature of 2.2076. SGLang serves it as `Qwen3_5ForConditionalGeneration` and writes the readout into the language model head rows of the code tokens, so the probabilities come out of the regular scoring path.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "34%"}} />
<col style={{width: "66%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Property</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Checkpoint</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/perplexity-ai/pplx-decider-v1-27b">perplexity-ai/pplx-decider-v1-27b</a> (BF16)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>Base model</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.8-27B, one epoch of supervised fine-tuning on decision data</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Question types</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>choice</code> (up to 255 options), <code>noul</code> (yes or no), <code>score</code> (up to 10 levels)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>Inputs</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Text or JSON state, optional images</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Prompt length</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Trained on question prompts of up to 8,192 tokens, and the backbone accepts 262,144</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>License</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Apache-2.0</td>
</tr>
</tbody>
</table>
Perplexity reports 85.71% overall accuracy across 11 public benchmarks for this checkpoint, against 74.76% for Qwen3.8-27B, measured through the Perplexity API. The per-benchmark table is on the model card. [§3](#3-benchmarks) measures accuracy and speed on SGLang.
**Resources:** [HuggingFace](https://huggingface.co/perplexity-ai/pplx-decider-v1-27b), [reference inference code](https://huggingface.co/perplexity-ai/pplx-decider-v1-27b/blob/main/inference.py), [Decision models in SGLang](/docs/supported-models/decision_models).
## 2. Configuration Tips
- **Route.** Send requests to `/v1/systemone`, the System One API the checkpoint was trained for. `/v1/decisions` refuses this checkpoint because its labels and prompt are not the ones the readout learned, and chat or generation requests are not meaningful without a language model head.
- **Clients.** Clients of the System One API, including the TypeSafe SDKs, work by pointing their base URL at the server. See [System One compatible API](/docs/supported-models/decision_models#system-one-compatible-api) for the request and response reference.
- **Images.** Pass each image in `images` as base64 bytes, a base64 data URL, or an `http(s)` URL. They precede the text in every question, as in the checkpoint's reference code. The checkpoint's processor resizes every image to between 65,536 and 262,144 pixels, so one image costs 64 to 256 prompt tokens: a 1024×768 screenshot is 234 tokens, and a 4K one is no more expensive than 1080p (252). Crop to the region that matters rather than sending a larger image.
- **What a decision costs.** Each question is its own prompt: an ~84-token wrapper (system message, chat template, answer instructions), the state, the question, and its options, read in one prefill pass. Several questions in one request are scored together, but each re-reads the whole state, so the cost of a request is roughly *questions × state tokens*.
- **Prefix cache.** On this hybrid model the prefix cache resumes from where an earlier prompt ended, because that is where the Gated DeltaNet state is saved, not from any shared prefix. An identical request measured 215 ms cold and 56 ms cached on GB300, but a different question about the same state, and the other questions in the same request, reused none of the state. The GB300 recipe therefore turns the cache off with `--disable-radix-cache`: unique-state traffic loses no reuse, each request holds one state slot instead of five, and the per-request state bookkeeping goes away, which took a short question from 59 to 51 ms and an image request from 126 to 87 ms with the attention kernel held fixed. Turn it back on only if your traffic repeats identical requests, the same state and the same question, often enough to matter.
- **Prompt length.** The model was fine-tuned on question prompts of up to 8,192 tokens, and the reference code refuses longer ones. SGLang serves the backbone's full 262,144-token context, so keep states within the trained length. A longer prompt is answered, but its accuracy is untested.
- **Throughput is compute-bound.** A decision is pure prefill, and one GB300 reads about 27,000 prompt tokens per second in BF16 once batches are full: about 71 requests per second for a 382-token question, and 3.3 for an 8K-token state. Past that point, more concurrency only adds queueing. The tables in [§3](#3-benchmarks) map concurrency to latency, so pick the highest concurrency that fits your latency budget.
- **Memory.** The weights take 52 GB, and the default `--mem-fraction-static` hands the rest of the GPU to the KV and Gated DeltaNet state pools. The startup log line `max_running_requests is capped to ... by the mamba state cache` is expected: on GB300, each request reserves five state slots with the prefix cache on (a cap of 118) and one with it off (593). Even the lower cap only binds for questions shorter than about 140 tokens, because a prefill batch fills its 16,384-token chunk first. `--mamba-ssm-dtype bfloat16` halves the state pool but measured no faster, so keep the checkpoint's FP32 state.
- **GB300 kernels.** With the prefix cache off, SGLang picks the TRT-LLM attention kernel with 64-token pages on GB300 (Triton with 1-token pages while the cache is on), and that kernel is where the long-state gain comes from: an 8K-token state drops from 377 to 325 ms and saturates at 3.30 instead of 2.87 requests per second, the same as forcing `--attention-backend trtllm_mha --page-size 64` with the cache on. FlashInfer's Gated DeltaNet prefill (`--linear-attn-prefill-backend flashinfer`) adds another 4-8% on text, taking the 8K-token state to 300 ms and 3.56 requests per second, but it made image requests slower (101 vs 87 ms alone, 9% less throughput), so the recipe leaves it to the Playground for text-only traffic. A larger prefill chunk (`--chunked-prefill-size 32768`) did not help: one 16,384-token chunk already keeps the GPU busy.
- **More GPUs.** The model fits one GPU, so add throughput with data-parallel replicas: `--dp-size 2` on two GB300s served 139 short questions per second (1.96× one GPU) and 6.5 8K-token states per second, and because the questions of one request are spread over the replicas, it also cut four questions on a 1K-token state from 179 to 104 ms. The extra dispatch hop added about 15 ms to a lone short question. Tensor parallelism only pays off for latency on long states: `--tp 2` took an 8K-token state from 325 to 208 ms, but delivered less throughput than two replicas and made a short question slower (68 ms), because the all-reduce dominates small batches. SGLang Model Gateway does not route `/v1/systemone` yet, so to spread load over several servers instead of `--dp-size`, use a plain HTTP load balancer.
- **FP8.** `--quantization fp8` quantizes the BF16 weights at load time and measured about 20% more throughput on GB300 (84 vs 70 requests per second for short questions), with the same Belebele and WinoGrande scores. But its probabilities moved by 0.010 on average and up to 0.20 against the reference implementation, four times the drift of any BF16 configuration, and 0.9% of answers changed their top option. The calibrated probabilities are this model's output, so the recipes stay in BF16, and the Playground offers FP8 for workloads that only use the top choice.
- **Reproducibility.** Two launches of the same recipe returned bit-identical probabilities for the same requests at the same concurrency. Against the checkpoint's reference implementation, the largest probability difference per question averages 0.003 (see [§3](#3-benchmarks)), because batch composition and kernels differ, and near-ties at 0.500 can flip.
## 3. Benchmarks
All numbers in this section come from one NVIDIA GB300 (288 GB) at SGLang commit `70f0b7351e`, with checkpoint revision `5117a6c`, served with the GB300 recipe above unless a row says otherwise.
**Method.** A closed-loop client keeps a fixed number of `/v1/systemone` requests in flight. Every request carries a unique random state, so requests share nothing but the system message, and each concurrency level starts after 8 warmup requests and a cache flush. Latency is the median end-to-end time of a request, and prompt tokens come from each response's `usage.input_tokens`. Four request shapes cover the common cases: one four-option choice question about a 256-token state, four questions (two choice, one yes or no, one score) about a 1,024-token state, one question about an 8,192-token state, and one question about a 1024×768 JPEG with a 64-token state.
### 3.1 Speed
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "36%"}} />
<col style={{width: "13%"}} />
<col style={{width: "17%"}} />
<col style={{width: "17%"}} />
<col style={{width: "17%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Request</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Concurrent</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Median latency</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Requests/s</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Prompt tokens/s</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Short question, 382 tokens</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>51 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>19.4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>7,413</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>103 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>32.8</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>12,525</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>16</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>267 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>59.0</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>22,531</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>882 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>70.8</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>27,039</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>256</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>3,521 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>71.1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>27,164</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Four questions on a 1K-token state, 4,540 tokens</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>179 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>5.6 (22 decisions/s)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>25,291</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>8</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>1,321 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>6.0 (24 decisions/s)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>27,332</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>32</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>5,206 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>6.0 (24 decisions/s)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>27,208</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>8K-token state, 8,224 tokens</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>325 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>3.08</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>25,297</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1,220 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>3.26</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>26,785</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>16</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>4,796 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>3.30</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>27,175</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>One 1024×768 image, 429 tokens</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>87 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>11.5</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>4,945</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>16</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>414 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>37.5</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>16,078</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1,170 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>52.6</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>22,572</td>
</tr>
</tbody>
</table>
One GB300 tops out at about 27,000 prompt tokens per second for every text shape, so requests per second follow from prompt length. Latency rises with concurrency once that ceiling is reached, so choose the concurrency from the latency you can accept. A decision has no output tokens, so in the benchmark card above TTFT is the whole request latency, and TPOT and interactivity do not apply.
### 3.2 Accuracy and parity
Both sets go through `/v1/systemone` as one `choice` question per item. Belebele sends the passage as the state, the question as the instructions, and the four answers as options `1` to `4`. WinoGrande sends the sentence as the state, asks which option fills the blank, and offers the two candidates. The reference row runs the same items through the checkpoint's own `DecisionModel` one at a time.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "50%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Implementation</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Belebele (eng_Latn, 900)</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>WinoGrande (xl dev, 1,267)</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang, GB300 recipe</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>96.67%</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>84.21%</strong></td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>SGLang, no extra flag</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>96.67%</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>84.37%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Checkpoint's reference <code>DecisionModel</code> (Transformers, BF16, GB300)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>96.67%</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>84.37%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Perplexity, model card (Perplexity API, its own prompt conversion)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>94.00%</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>83.30%</td>
</tr>
</tbody>
</table>
Against the reference implementation, the recipe picks the same option on 2,161 of 2,167 items, and each of the six differences is a near-tie that one side scores 0.500 against 0.500 and the other 0.486 against 0.514. Per item, the largest probability difference averages 0.003 (99th percentile 0.024, maximum 0.054). Perplexity's figures come from a different prompt conversion, so they are context rather than a target.
### 3.3 What else was measured
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "33%"}} />
<col style={{width: "45%"}} />
<col style={{width: "22%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Change from the GB300 recipe</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Measured effect</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Verdict</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Prefix cache on (no extra flag)</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Short question 58 ms alone, 8K-token state 377 ms alone and 2.87 requests/s saturated (recipe: 51 ms, 325 ms, 3.30)</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Only for traffic that repeats identical requests</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}><code>--linear-attn-prefill-backend flashinfer</code></td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Text 4-8% faster (8K-token state 300 ms, 3.56 requests/s), images slower (101 ms alone, 9% less throughput)</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Text-only traffic</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}><code>--dp-size 2</code> (2 GPUs)</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>139 short questions/s and 6.5 8K-token states/s, four questions on a 1K-token state in 104 ms, a lone short question 15 ms slower</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Scale-out</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}><code>--tp 2</code> (2 GPUs)</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>8K-token state 208 ms alone, 124 short questions/s saturated, a lone short question in 68 ms</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Long states with a latency target</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}><code>--quantization fp8</code></td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>About 20% more throughput, same accuracy, probabilities shifted by 0.010 on average and up to 0.20</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Only when you use the top choice alone</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}><code>--chunked-prefill-size 32768 --max-prefill-tokens 32768</code></td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>No gain (within 4%)</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Keep the default</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}><code>--mamba-ssm-dtype bfloat16</code></td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>No gain, though it halves the state pool</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Keep the default</td>
</tr>
</tbody>
</table>
<Accordion title="Check accuracy on your deployment (Python)">
```python Example
import asyncio
import aiohttp
from datasets import load_dataset
URL = "http://localhost:30000/v1/systemone"
async def main():
rows = load_dataset("facebook/belebele", "eng_Latn", split="test")
limit = asyncio.Semaphore(32)
async def ask(session, row):
body = {
"model": "perplexity-ai/pplx-decider-v1-27b",
"state": row["flores_passage"],
"questions": {
"answer": {
"type": "choice",
"instructions": row["question"],
"criteria": {str(k): row[f"mc_answer{k}"] for k in range(1, 5)},
}
},
}
async with limit, session.post(URL, json=body) as response:
response.raise_for_status()
answer = (await response.json())["answers"]["answer"]
return answer["choice"] == row["correct_answer_num"]
async with aiohttp.ClientSession() as session:
correct = await asyncio.gather(*(ask(session, row) for row in rows))
print(f"Belebele eng_Latn: {sum(correct)}/{len(correct)} = {100 * sum(correct) / len(correct):.2f}%")
asyncio.run(main())
```
</Accordion>
<Accordion title="Example Output">
```text Output
Belebele eng_Latn: 870/900 = 96.67%
```
</Accordion>
<Accordion title="Measure speed on your deployment (Python)">
Run it as `python bench.py <state words> <concurrency> <requests>`. With 270 words, a request is 378 tokens, close to the short shape above.
```python Example
import asyncio
import random
import statistics
import sys
import time
import aiohttp
BASE = "http://localhost:30000"
STATE_WORDS, CONCURRENCY, REQUESTS = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3])
WORDS = ("customer payment refund invoice order delivery account login error update server outage "
"contract price plan report urgent today please help the a and with for from to of in on").split()
QUESTION = {
"type": "choice",
"instructions": "Which team should handle this request?",
"criteria": {"billing": "Charges and refunds", "technical_support": "Integration errors",
"sales": "Questions about buying a product", "legal": "Contracts and liability"},
}
def body(rng):
# A unique random state per request, so no request reuses another's cache.
state = " ".join(rng.choices(WORDS, k=STATE_WORDS))
return {"model": "perplexity-ai/pplx-decider-v1-27b", "state": state, "questions": {"q": QUESTION}}
async def main():
rng = random.Random(0)
bodies = [body(rng) for _ in range(REQUESTS)]
latencies, tokens = [], 0
async with aiohttp.ClientSession() as session:
for item in [body(rng) for _ in range(8)]: # warmup
async with session.post(f"{BASE}/v1/systemone", json=item) as response:
response.raise_for_status()
async with session.post(f"{BASE}/flush_cache") as response:
await response.read()
queue = iter(bodies)
async def worker():
nonlocal tokens
for item in queue:
start = time.perf_counter()
async with session.post(f"{BASE}/v1/systemone", json=item) as response:
response.raise_for_status()
tokens += (await response.json())["usage"]["input_tokens"]
latencies.append(time.perf_counter() - start)
start = time.perf_counter()
await asyncio.gather(*(worker() for _ in range(CONCURRENCY)))
elapsed = time.perf_counter() - start
print(f"{REQUESTS / elapsed:.1f} req/s, {tokens / elapsed:.0f} prompt tok/s, "
f"P50 {1000 * statistics.median(latencies):.1f} ms, {tokens / REQUESTS:.0f} tokens/request")
asyncio.run(main())
```
</Accordion>
<Accordion title="Example Output">
```text Output
$ python bench.py 270 1 64
19.6 req/s, 7393 prompt tok/s, P50 51.0 ms, 378 tokens/request
$ python bench.py 270 64 512
68.4 req/s, 25865 prompt tok/s, P50 899.0 ms, 378 tokens/request
```
</Accordion>
## 4. Advanced Usage
### 4.1 Text Decisions
<Accordion title="Text Decision Example (Python)">
```python Example
import requests
response = requests.post(
"http://localhost:30000/v1/systemone",
json={
"model": "perplexity-ai/pplx-decider-v1-27b",
"state": "My Stripe integration keeps failing. I'm losing sales. Please help ASAP.",
"questions": {
"routing": {
"type": "choice",
"instructions": "Which team should handle this request?",
"criteria": {
"billing": "Charges and refunds",
"technical_support": "Integration errors",
"sales": "Questions about buying a product",
},
},
"urgency": {"type": "noul", "instructions": "Does this message express urgency?"},
},
},
timeout=60,
)
response.raise_for_status()
for name, answer in response.json()["answers"].items():
print(name, answer)
```
</Accordion>
<Accordion title="Example Output">
```text Output
routing {'type': 'choice', 'choice': 'technical_support', 'confidence': 0.987854396908662, 'probabilities': {'billing': 0.004840538662952211, 'technical_support': 0.9919029312724412, 'sales': 0.0032565300646064375}}
urgency {'type': 'noul', 'noul': 0.9929980993021188}
```
</Accordion>
### 4.2 Image Decisions
The example output below came from a mostly red test image.
<Accordion title="Image Decision Example (Python)">
```python Example
import base64
from pathlib import Path
import requests
image = base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii")
response = requests.post(
"http://localhost:30000/v1/systemone",
json={
"model": "perplexity-ai/pplx-decider-v1-27b",
"state": "Look at the supplied image.",
"images": [image],
"questions": {
"color": {
"type": "choice",
"instructions": "What is the dominant color?",
"criteria": {"red": "Red", "green": "Green", "blue": "Blue", "other": "Another color"},
}
},
},
timeout=60,
)
response.raise_for_status()
print(response.json()["answers"]["color"])
```
</Accordion>
<Accordion title="Example Output">
```text Output
{'type': 'choice', 'choice': 'red', 'confidence': 0.9498445161931509, 'probabilities': {'red': 0.9623833871448632, 'green': 0.009006826727044857, 'blue': 0.00926546931819573, 'other': 0.01934431680989631}}
```
</Accordion>
@@ -0,0 +1,498 @@
---
title: PPLX-Decider-v1.1-27B
description: "Deploy PPLX-Decider-v1.1-27B with SGLang. Perplexity's updated decision model reads the whole prompt in its full-attention layers and returns calibrated probabilities for choice, yes or no, and score questions on /v1/systemone, on one NVIDIA GB300 or H200 in BF16."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. PPLX-Decider-v1.1 support lands in [#42645](https://github.com/sgl-project/sglang/pull/42645), after v0.5.21, so until a release includes it, install a [nightly build](/docs/get-started/install#nightly-builds) built after that merge.
<Tabs>
<Tab title="Python (pip / uv)">
```bash Command
pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
```
Then run the **Python** output of the command panel below in that environment.
</Tab>
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:dev
```
The image is multi-arch, so the same tag runs on x86 H200 hosts and Arm GB300 hosts. For how to launch it, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
Pick your GPU to generate the launch command. The model runs on one GPU in BF16 and needs no flag: SGLang reads the checkpoint's `decision_config.json`, runs the full-attention layers over the whole prompt as the model was trained, and turns off the radix cache and chunked prefill, which would show those layers only part of the prompt.
import { Deployment } from "/src/snippets/_deployment.jsx"
import { config } from "/src/snippets/configs/perplexity-ai/pplx-decider-v1.1-27b.jsx"
import { benchmarks } from "/src/snippets/configs/perplexity-ai/pplx-decider-v1.1-27b-benchmarks.jsx"
<Deployment config={config} benchmarks={benchmarks} />
<Note>
The GB300 recipe was measured end to end on one GB300 at [#42645](https://github.com/sgl-project/sglang/pull/42645): four request shapes from 1 to 256 concurrent requests, Belebele and WinoGrande accuracy, and per-question parity with the checkpoint's own reference implementation. The H200 command is the same and has not been run yet. [§3](#3-benchmarks) has the method and the full tables.
</Note>
## Playground
The Playground layers SGLang features on top of the recipe above. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end. The [Configuration Tips](#2-configuration-tips) say which overrides were measured on GB300. Tensor parallelism times data-parallel replicas must fit the GPUs on one node.
import { Playground } from "/src/snippets/_playground.jsx"
<Playground config={config} />
## 1. Model Introduction
**PPLX-Decider-v1.1-27B** is the successor to [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B), Perplexity's decision model fine-tuned from [Qwen3.8-27B](/cookbook/autoregressive/Qwen/Qwen3.8-27B) under the Apache-2.0 license. Like v1, it replaces the language model head with a readout over 255 answer codes and answers choice, yes or no (`noul`), and score questions about a text state and optional images with a probability for every option, without generating text.
Two things changed. The 16 full-attention layers no longer use a causal mask, so every token attends to the whole prompt, while the 48 Gated DeltaNet layers stay causal. And training grew from 73,000 to 626,033 rows, most of them from [tasksource](https://github.com/sileod/tasksource). The prompt format, answer codes, and readout layout are the same as v1's, but the calibrated temperature is 1.0087 instead of 2.2076, so probabilities from the two versions are not interchangeable.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "30%"}} />
<col style={{width: "70%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Property</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Checkpoint</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b">perplexity-ai/pplx-decider-v1.1-27b</a> (BF16)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Base model</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.8-27B, one epoch of supervised fine-tuning on 626,033 decision rows (530,103 from tasksource)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Attention</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Noncausal in the 16 full-attention layers, causal in the 48 Gated DeltaNet layers</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Question types</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}><code>choice</code> (up to 255 options), <code>noul</code> (yes or no), <code>score</code> (up to 10 levels)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Inputs</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Text or JSON state, optional images</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Prompt length</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Trained on question prompts of up to 8,192 tokens, and the backbone accepts 262,144</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>License</td>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Apache-2.0</td>
</tr>
</tbody>
</table>
Perplexity reports a Decision Index of 61.56 for v1.1, up from 56.4 for v1:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "34%"}} />
<col style={{width: "18%"}} />
<col style={{width: "24%"}} />
<col style={{width: "24%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Decision Index category</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Jev</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>PPLX-Decider-v1-27B</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>PPLX-Decider-v1.1-27B</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Knowledge</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>51.4</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>40.9</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>48.18</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Language</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>62.0</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>63.5</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>69.45</strong></td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Retrieval</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>55.4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>54.9</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>61.26</strong></td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Tools</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>75.1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>79.3</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>78.88</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Arts</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>37.7</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>39.4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>44.66</strong></td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Overall</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>57.9</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>56.4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>61.56</strong></td>
</tr>
</tbody>
</table>
**Resources:** [HuggingFace](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b), [reference model code](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b/blob/main/source/src/autojev/model.py), [Decision models in SGLang](/docs/supported-models/decision_models).
## 2. Configuration Tips
- **Route.** Send requests to `/v1/systemone`, the System One API the checkpoint was trained for. `/v1/decisions` refuses this checkpoint because its labels and prompt are not the ones the readout learned, and chat or generation requests are not meaningful without a language model head.
- **Clients.** Clients of the System One API, including the TypeSafe SDKs, work by pointing their base URL at the server. See [System One compatible API](/docs/supported-models/decision_models#system-one-compatible-api) for the request and response reference.
- **Images.** Pass each image in `images` as base64 bytes, a base64 data URL, or an `http(s)` URL. They precede the text in every question. The processor resizes every image to between 65,536 and 262,144 pixels, so one image costs 64 to 256 prompt tokens.
- **Noncausal attention is automatic.** SGLang reads `attention_mode: noncausal_full_attention` from `decision_config.json` and runs the full-attention layers as encoder layers, which the TRT-LLM, FlashInfer, and Triton attention backends support. It also turns off the radix cache and chunked prefill, logging `Radix cache and chunked prefill are disabled for a decision checkpoint with noncausal full attention`: a cached prefix or a prefill chunk would see only part of a prompt whose every position depends on the rest. These two settings override any flag you pass. A prompt longer than the 16,384-token prefill budget is still read in one pass: an 18,585-token prompt matched the reference implementation to four decimal places on GB300.
- **No prefix reuse.** Because every layer output depends on the whole prompt, nothing carries over between requests, not even an identical one. The cost of a request is roughly *questions × prompt tokens*, as on v1.
- **Prompt length.** Like v1, the model was trained on question prompts of up to 8,192 tokens, and the reference code refuses longer ones. SGLang serves longer prompts in one pass, but their accuracy is untested.
- **Attention backend.** On GB300, Auto resolves to the TRT-LLM kernel with 64-token pages. `--attention-backend flashinfer` and `--attention-backend triton` matched the reference implementation as closely, with the same top option on 99.8% of the accuracy items. The H200 path has not been run yet.
- **Speed.** One GB300 reads about 27,000 prompt tokens per second once batches are full, the same as v1. Measured back to back on the same GPU against v1's recipe, v1.1 was within 2% on short questions and four-question requests, and an 8K-token state saturated 6% lower (3.13 against 3.32 requests per second), because noncausal attention computes the full attention matrix in its 16 full-attention layers instead of half of it.
- **More GPUs.** The backbone and prompt sizes are v1's, so the [v1 page's scaling measurements](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B#2-configuration-tips) are the reference: data-parallel replicas (`--dp-size N`) for throughput, tensor parallelism only for latency on long prompts. They were not repeated for v1.1.
- **Upgrading from v1.** Re-check any thresholds you tuned on v1 probabilities: v1.1 has a different temperature and changed its top answer on 8% of the Belebele and WinoGrande items.
## 3. Benchmarks
All numbers in this section come from one NVIDIA GB300 (288 GB) at [#42645](https://github.com/sgl-project/sglang/pull/42645), with checkpoint revision `3b45dea`, served with the GB300 recipe above unless a row says otherwise. The method and request shapes are those of the [v1 page](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B#3-benchmarks), whose speed script works here unchanged.
### 3.1 Speed
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "36%"}} />
<col style={{width: "13%"}} />
<col style={{width: "17%"}} />
<col style={{width: "17%"}} />
<col style={{width: "17%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Request</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Concurrent</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Median latency</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Requests/s</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Prompt tokens/s</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>Short question, 382 tokens</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>53 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>18.9</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>7,203</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>106 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>37.6</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>14,362</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>16</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>275 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>57.4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>21,909</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>931 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>67.5</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>25,770</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>256</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>3,489 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>71.1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>27,139</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Four questions on a 1K-token state, 4,540 tokens</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>183 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>5.5 (22 decisions/s)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>24,782</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>8</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>1,315 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>6.0 (24 decisions/s)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>27,398</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>32</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>4,882 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>6.1 (24 decisions/s)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>27,649</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>8K-token state, 8,224 tokens</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>334 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>2.99</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>24,620</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1,280 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>3.12</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>25,682</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>16</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>5,101 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>3.13</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>25,726</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>One 1024×768 image, 429 tokens</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>118 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>8.3</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>3,541</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>16</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>466 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>33.6</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>14,398</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1,197 ms</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>49.9</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>21,416</td>
</tr>
</tbody>
</table>
### 3.2 Accuracy and parity
Both sets go through `/v1/systemone` as one `choice` question per item, with the same conversion as the v1 page. The reference rows run the same items through the checkpoint's own `DecisionModel` one at a time.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "50%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Implementation</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Belebele (eng_Latn, 900)</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>WinoGrande (xl dev, 1,267)</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang, GB300 recipe (TRT-LLM attention)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>97.67%</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>92.42%</strong></td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>SGLang, <code>--attention-backend flashinfer</code></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>97.67%</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>92.58%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang, <code>--attention-backend triton</code></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>97.67%</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>92.42%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>Checkpoint's reference <code>DecisionModel</code> (Transformers, BF16, GB300)</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>97.67%</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>92.50%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.02)"}}>v1.1 served with causal attention, as SGLang did before #42645</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>97.00%</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>88.87%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", textAlign: "left", backgroundColor: "rgba(255,255,255,0.05)"}}>PPLX-Decider-v1-27B on SGLang, for comparison</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>96.67%</td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>84.21%</td>
</tr>
</tbody>
</table>
Against the reference implementation, the recipe picks the same option on 2,164 of 2,167 items, and each of the three differences is a near-tie within 0.031 of an even split. Per item, the largest probability difference averages 0.002 (99th percentile 0.030, maximum 0.089). Prompts of 11,099 and 18,585 tokens, an image question, and a score question also matched the reference, the largest difference being 0.015 on the score question. Served with causal attention instead, v1.1 changed its top answer on 101 of the 2,167 items and lost 3.6 points on WinoGrande.
<Accordion title="Check accuracy on your deployment (Python)">
```python Example
import asyncio
import aiohttp
from datasets import load_dataset
URL = "http://localhost:30000/v1/systemone"
async def main():
rows = load_dataset("facebook/belebele", "eng_Latn", split="test")
limit = asyncio.Semaphore(32)
async def ask(session, row):
body = {
"model": "perplexity-ai/pplx-decider-v1.1-27b",
"state": row["flores_passage"],
"questions": {
"answer": {
"type": "choice",
"instructions": row["question"],
"criteria": {str(k): row[f"mc_answer{k}"] for k in range(1, 5)},
}
},
}
async with limit, session.post(URL, json=body) as response:
response.raise_for_status()
answer = (await response.json())["answers"]["answer"]
return answer["choice"] == row["correct_answer_num"]
async with aiohttp.ClientSession() as session:
correct = await asyncio.gather(*(ask(session, row) for row in rows))
print(f"Belebele eng_Latn: {sum(correct)}/{len(correct)} = {100 * sum(correct) / len(correct):.2f}%")
asyncio.run(main())
```
</Accordion>
<Accordion title="Example Output">
```text Output
Belebele eng_Latn: 879/900 = 97.67%
```
</Accordion>
## 4. Advanced Usage
### 4.1 Text Decisions
<Accordion title="Text Decision Example (Python)">
```python Example
import requests
response = requests.post(
"http://localhost:30000/v1/systemone",
json={
"model": "perplexity-ai/pplx-decider-v1.1-27b",
"state": "My Stripe integration keeps failing. I'm losing sales. Please help ASAP.",
"questions": {
"routing": {
"type": "choice",
"instructions": "Which team should handle this request?",
"criteria": {
"billing": "Charges and refunds",
"technical_support": "Integration errors",
"sales": "Questions about buying a product",
},
},
"urgency": {"type": "noul", "instructions": "Does this message express urgency?"},
},
},
timeout=60,
)
response.raise_for_status()
for name, answer in response.json()["answers"].items():
print(name, answer)
```
</Accordion>
<Accordion title="Example Output">
```text Output
routing {'type': 'choice', 'choice': 'technical_support', 'confidence': 0.9977658531751561, 'probabilities': {'billing': 0.0014030901008239556, 'technical_support': 0.9985105687834377, 'sales': 8.634111573847184e-05}}
urgency {'type': 'noul', 'noul': 0.9995671977710632}
```
</Accordion>
### 4.2 Image Decisions
The example output below came from a 512×512 solid red test image.
<Accordion title="Image Decision Example (Python)">
```python Example
import base64
from pathlib import Path
import requests
image = base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii")
response = requests.post(
"http://localhost:30000/v1/systemone",
json={
"model": "perplexity-ai/pplx-decider-v1.1-27b",
"state": "Look at the supplied image.",
"images": [image],
"questions": {
"color": {
"type": "choice",
"instructions": "What is the dominant color?",
"criteria": {"red": "Red", "green": "Green", "blue": "Blue", "other": "Another color"},
}
},
},
timeout=60,
)
response.raise_for_status()
print(response.json()["answers"]["color"])
```
</Accordion>
<Accordion title="Example Output">
```text Output
{'type': 'choice', 'choice': 'red', 'confidence': 0.9989117512916561, 'probabilities': {'red': 0.9991838134687421, 'green': 6.743375714875418e-05, 'blue': 8.12085765657198e-05, 'other': 0.0006675441975434462}}
```
</Accordion>
@@ -1,126 +0,0 @@
---
title: pplx-decider-v1-27b
description: "Deploy pplx-decider-v1-27b with SGLang. Perplexity's decision model, fine-tuned from Qwen3.8-27B, answers choice, yes or no, and score questions on one NVIDIA H200."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
Until a release includes pplx-decider support, use a [nightly build](/docs/get-started/install#nightly-builds). The nightly image carries it:
```bash Command
docker pull lmsysorg/sglang:dev
```
</Accordion>
The recipe below serves the model in BF16 on one H200 with no extra flag. SGLang reads the checkpoint's `decision_config.json`, loads its decision readout, and answers on `/v1/systemone` with the prompt, answer codes, and calibrated temperature the model was trained with.
import { Deployment } from "/src/snippets/_deployment.jsx"
import { config } from "/src/snippets/configs/perplexity-ai/pplx-decider-v1-27b.jsx"
<Deployment config={config} />
## 1. Model Introduction
**pplx-decider-v1-27b** is a decision model from Perplexity, fine-tuned from Qwen3.8-27B under the Apache-2.0 license. It replaces the language model head with a readout over 255 answer codes, and answers choice, yes or no (`noul`), and score questions about a text state and optional images with a probability for every option, without generating text.
**Resources:** [HuggingFace](https://huggingface.co/perplexity-ai/pplx-decider-v1-27b).
## 2. Configuration Tips
- **Route.** Send requests to `/v1/systemone`, the System One API the checkpoint was trained for. `/v1/decisions` refuses this checkpoint because its labels and prompt are not the ones the readout learned, and chat or generation requests are not meaningful without a language model head.
- **Clients.** Clients of the System One API, including the TypeSafe SDKs, work by pointing their base URL at the server. See [System One compatible API](/docs/supported-models/decision_models#system-one-compatible-api) for the request and response reference.
- **Images.** Pass each image in `images` as base64 bytes or a URL. They precede the text in every question, as in the checkpoint's reference code.
## 3. Advanced Usage
### 3.1 Text Decisions
<Accordion title="Text Decision Example (Python)">
```python Example
import requests
response = requests.post(
"http://localhost:30000/v1/systemone",
json={
"model": "perplexity-ai/pplx-decider-v1-27b",
"state": "My Stripe integration keeps failing. I'm losing sales. Please help ASAP.",
"questions": {
"routing": {
"type": "choice",
"instructions": "Which team should handle this request?",
"criteria": {
"billing": "Charges and refunds",
"technical_support": "Integration errors",
"sales": "Questions about buying a product",
},
},
"urgency": {"type": "noul", "instructions": "Does this message express urgency?"},
},
},
timeout=60,
)
response.raise_for_status()
for name, answer in response.json()["answers"].items():
print(name, answer)
```
</Accordion>
<Accordion title="Example Output">
```text Output
routing {'type': 'choice', 'choice': 'technical_support', 'confidence': 0.987854396908662, 'probabilities': {'billing': 0.004840538662952211, 'technical_support': 0.9919029312724412, 'sales': 0.0032565300646064375}}
urgency {'type': 'noul', 'noul': 0.9929980993021188}
```
</Accordion>
### 3.2 Image Decisions
The example output below came from a mostly red test image.
<Accordion title="Image Decision Example (Python)">
```python Example
import base64
from pathlib import Path
import requests
image = base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii")
response = requests.post(
"http://localhost:30000/v1/systemone",
json={
"model": "perplexity-ai/pplx-decider-v1-27b",
"state": "Look at the supplied image.",
"images": [image],
"questions": {
"color": {
"type": "choice",
"instructions": "What is the dominant color?",
"criteria": {"red": "Red", "green": "Green", "blue": "Blue", "other": "Another color"},
}
},
},
timeout=60,
)
response.raise_for_status()
print(response.json()["answers"]["color"])
```
</Accordion>
<Accordion title="Example Output">
```text Output
{'type': 'choice', 'choice': 'red', 'confidence': 0.9498445161931509, 'probabilities': {'red': 0.9623833871448632, 'green': 0.009006826727044857, 'blue': 0.00926546931819573, 'other': 0.01934431680989631}}
```
</Accordion>
+1 -1
View File
@@ -190,7 +190,7 @@ metatags:
<Card
title="Perplexity"
mode="card"
href="/cookbook/autoregressive/Perplexity/pplx-decider-v1-27b"
href="/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B"
img="/cards/logos/perplexity.png"
/>
</CardGroup>
+6 -1
View File
@@ -17,6 +17,10 @@
"source": "/cookbook",
"destination": "/cookbook/intro"
},
{
"source": "/cookbook/autoregressive/Perplexity/pplx-decider-v1-27b",
"destination": "/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B"
},
{
"source": "/whl",
"destination": "https://sgl-project.github.io/whl/",
@@ -1481,7 +1485,8 @@
{
"group": "Perplexity",
"pages": [
"cookbook/autoregressive/Perplexity/pplx-decider-v1-27b"
"cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B",
"cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B"
]
}
]
@@ -159,7 +159,7 @@ The request fails with HTTP 400 for:
- a server launched with `--enable-mis` or `--dllm-algorithm`, or one that uses a built-in conversation template, whether named with `--chat-template`, loaded from a JSON template file, or inferred from the model path
- a model whose chat route uses a built-in encoder instead of a chat template, a tokenizer that does not encode the rendered chat text back to the same ids, or a model that is not a generation model
- a chat template that raises an error for the question message, or a server launched with `--skip-tokenizer-init`
- a decision checkpoint such as [pplx-decider-v1-27b](/cookbook/autoregressive/Perplexity/pplx-decider-v1-27b), which answers through `/v1/systemone`
- a decision checkpoint such as [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B), which answers through `/v1/systemone`
Errors about one question name its id or its position in `questions`.
@@ -207,7 +207,7 @@ These models mix full and linear attention. Answer probabilities can move by up
## System One compatible API
`POST /v1/systemone` serves the same decisions in the request and response shape of the System One API, with a `state`, a map of `noul`, `choice`, and `score` questions keyed by your ids, and one answer per id. Clients written for that API, including the official TypeSafe SDKs, get decisions by pointing their base URL at the server, with the exceptions listed below. The route uses the same rendering, label checks, and scoring as `/v1/decisions`, and refuses the same servers, chat templates, and prompts, listed in [Errors](#errors). Its request fields are checked as described below. A decision checkpoint such as [pplx-decider-v1-27b](/cookbook/autoregressive/Perplexity/pplx-decider-v1-27b) instead gets the prompt, answer codes, and calibrated temperature it was trained with.
`POST /v1/systemone` serves the same decisions in the request and response shape of the System One API, with a `state`, a map of `noul`, `choice`, and `score` questions keyed by your ids, and one answer per id. Clients written for that API, including the official TypeSafe SDKs, get decisions by pointing their base URL at the server, with the exceptions listed below. The route uses the same rendering, label checks, and scoring as `/v1/decisions`, and refuses the same servers, chat templates, and prompts, listed in [Errors](#errors). Its request fields are checked as described below. A decision checkpoint such as [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B) instead gets the prompt, answer codes, and calibrated temperature it was trained with. When its `decision_config.json` sets `attention_mode` to `noncausal_full_attention`, as [PPLX-Decider-v1.1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B) does, its full-attention layers read the whole prompt, and the server turns off the radix cache and chunked prefill, which would show those layers only part of it.
```bash Command
pip install typesafe-sdk
@@ -0,0 +1,38 @@
// PPLX-Decider-v1-27B per-cell benchmark numbers. See _deployment.jsx for the
// speed/accuracy schema.
//
// A decision is prefill-only: the answer is read from the logits at the end of
// the prompt, so TTFT is the whole request latency, there is no TPOT (and no
// derived interactivity), and throughput per GPU is prompt tokens/s per GPU
// (osl = 0). TTFT is the P50 of end-to-end /v1/systemone request time.
//
// GB300: one GPU, the cell's recipe (--disable-radix-cache), SGLang main @
// 70f0b7351e, checkpoint revision 5117a6c. Closed-loop client against
// /v1/systemone: every request carries a unique random state (only the system
// message is shared), 8 warmup requests, cache flushed before each level. isl
// is the mean prompt length per request: a 256-token state with one 4-option
// choice question (382), or an 8,192-token state with the same question
// (8,224). Section 3 of the page has the full tables and the four-question and image
// shapes.
//
// Accuracy through /v1/systemone with the page's own conversion: Belebele
// eng_Latn test (900) and WinoGrande xl validation (1,267). The checkpoint's
// reference DecisionModel scores 96.67 / 84.37 on the same items.
export const benchmarks = [
{
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
sglang_version: "main @ 70f0b7351e",
speed: [
{ workload: { dataset: "random", isl: 382, osl: 0, max_concurrency: 1, num_prompts: 64 },
ttft_ms: 51.4, tpot_ms: null, tokens_per_sec_per_gpu: 7413 },
{ workload: { dataset: "random", isl: 382, osl: 0, max_concurrency: 64, num_prompts: 256 },
ttft_ms: 881.9, tpot_ms: null, tokens_per_sec_per_gpu: 27039 },
{ workload: { dataset: "random", isl: 8224, osl: 0, max_concurrency: 1, num_prompts: 64 },
ttft_ms: 324.7, tpot_ms: null, tokens_per_sec_per_gpu: 25297 },
{ workload: { dataset: "random", isl: 8224, osl: 0, max_concurrency: 16, num_prompts: 64 },
ttft_ms: 4796.2, tpot_ms: null, tokens_per_sec_per_gpu: 27175 },
],
accuracy: { belebele_pct: 96.67, winogrande_pct: 84.21 },
notes: "Prefill-only model: TTFT is the full decision latency and throughput is prompt tokens/s/GPU. TPOT and interactivity do not apply.",
},
]
@@ -1,17 +1,33 @@
// Instantiated from cookbook-add-model/templates/config.jsx.tmpl.
// One recipe, verified on H200 in BF16. The decision readout needs no extra flag.
// Single `export const config` literal with no spreads/calls/IIFE (Mintlify re-evals at hydration).
//
// PPLX-Decider-v1-27B: a decision checkpoint (Qwen3.8-27B backbone + a 255-row
// readout) served on /v1/systemone. Every request is prefill-only, one pass
// per question with no decode, so there are no parser, speculative-decoding or
// PD-disaggregation knobs. BF16 is the only published precision.
//
// H200: the original recipe, verified by the model-support PR (#42183) on one
// H200 with no extra flag. GB300: measured end to end on main @ 70f0b7351e for
// speed, Belebele/WinoGrande accuracy, and per-item parity against the
// checkpoint's own reference DecisionModel (see the benchmarks file).
export const config = {
modelName: "pplx-decider-v1-27b",
supportedHardware: ["h200"],
modelName: "PPLX-Decider-v1-27B",
latencyPercentile: "P50",
supportedHardware: ["h200", "gb300"],
variants: [
{ id: "default", label: "Default", subtitle: "27B dense, 1 GPU" },
],
quantizations: [{ id: "bf16", label: "BF16" }],
strategies: [{ id: "balanced", label: "Balanced" }],
nodesOptions: [{ id: "single", label: "Single Node" }],
modelNames: {
"default|bf16": "perplexity-ai/pplx-decider-v1-27b",
},
placeholders: {
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
PORT: { target: "command", label: "Bind port", default: "30000" },
@@ -19,11 +35,99 @@ export const config = {
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
},
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/systemone \\
-H 'Content-Type: application/json' \\
-d '{"model":"{{MODEL_NAME}}","state":"My Stripe integration keeps failing. Please help ASAP.","questions":{"urgency":{"type":"noul","instructions":"Does this message express urgency?"}}}'`,
dockerImages: { h200: "lmsysorg/sglang:dev" },
// Accuracy through /v1/systemone on public sets the model card also reports,
// with this page's own conversion (see the page's section 3). Not comparable
// number-for-number with the card, which used Perplexity's converters.
accuracyLabels: [
["belebele_pct", "Belebele (eng_Latn, 900)", "%"],
["winogrande_pct", "WinoGrande (xl dev, 1267)", "%"],
],
// `lmsysorg/sglang:dev` is multi-arch (amd64 + arm64), so GB300 hosts pull
// the same tag as H200 hosts.
dockerImages: {
h200: "lmsysorg/sglang:dev",
gb300: "lmsysorg/sglang:dev",
},
github: { cookbookModel: "perplexity-ai/pplx-decider-v1-27b" },
// Every request is prefill-only (one pass per question, no decode), so the
// general axes that act on generation are omitted: parsers (no text output),
// speculative decoding and PD disaggregation (no decode phase). moe: dense
// model. CP / DP-Attention: no model-side support for this architecture.
// hicache: the prefix cache only resumes from where an earlier prompt ended
// (the Gated DeltaNet state is saved there), so a host-memory tier adds
// nothing to unique-state decision traffic.
playgroundFeatures: {
attention: {
knobs: [
{ id: "tp", label: "TP", values: [null, 1, 2, 4] },
],
},
flagSelects: [
{
// Replicas of the 1-GPU model behind one endpoint, the throughput lever.
id: "dp", title: "Data Parallel Replicas",
stripPrefixes: ["--dp-size"],
options: [
{ id: "1", label: "1" },
{ id: "2", label: "2", flags: ["--dp-size 2"] },
{ id: "4", label: "4", flags: ["--dp-size 4"] },
],
},
{
// Only an identical prompt can resume from the cache on this hybrid
// model. Off also frees the per-request state slots (S=1).
id: "prefixCache", title: "Prefix Cache",
stripPrefixes: ["--disable-radix-cache"],
options: [
{ id: "on", label: "On (reuses identical prompts)" },
{ id: "off", label: "Off (unique states)", flags: ["--disable-radix-cache"] },
],
},
{
// trtllm_mha needs SM100 and a 64-token page. With the prefix cache
// off, Auto already resolves to it on GB300.
id: "attnBackend", title: "Attention Backend",
stripPrefixes: ["--attention-backend", "--page-size"],
options: [
{ id: "auto", label: "Auto" },
{ id: "trtllm", label: "TRT-LLM MHA",
flags: ["--attention-backend trtllm_mha", "--page-size 64"],
disable: { hw: ["h200"] },
disableReason: "trtllm_mha is a Blackwell (SM100) kernel" },
{ id: "triton", label: "Triton", flags: ["--attention-backend triton"] },
],
},
{
id: "gdnPrefill", title: "Gated DeltaNet Prefill Kernel",
stripPrefixes: ["--linear-attn-prefill-backend"],
options: [
{ id: "auto", label: "Auto" },
{ id: "triton", label: "Triton", flags: ["--linear-attn-prefill-backend triton"] },
{ id: "flashinfer", label: "FlashInfer", flags: ["--linear-attn-prefill-backend flashinfer"] },
],
},
{
// Load-time FP8 of the BF16 checkpoint: ~20% more throughput on GB300,
// but it moves the calibrated probabilities (mean 0.010, max 0.20 vs
// the reference implementation), so it is never in a cell.
id: "weights", title: "Weight Precision",
stripPrefixes: ["--quantization"],
options: [
{ id: "bf16", label: "BF16 (checkpoint)" },
{ id: "fp8", label: "FP8 at load (shifts probabilities)", flags: ["--quantization fp8"] },
],
},
],
},
cells: [
{
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
@@ -35,5 +139,24 @@ export const config = {
"--port {{PORT}}",
],
},
{
// GB300 (SM103), one GPU. Prefix reuse on this hybrid model only happens
// for identical prompts, so the cache is off: each request needs one
// Gated DeltaNet state slot instead of five, and with the cache off the
// Qwen3.5 model hook resolves attention to trtllm_mha with 64-token
// pages (Triton with 1-token pages otherwise). Isolated A/B on main @
// 70f0b7351e against the no-flag recipe: 8K-token state 377 -> 325 ms at
// concurrency 1 and +15% saturated throughput, 380-token question 58 ->
// 51 ms, same Belebele/WinoGrande accuracy and reference parity.
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
verified: true,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--disable-radix-cache",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
],
}
@@ -0,0 +1,36 @@
// PPLX-Decider-v1.1-27B per-cell benchmark numbers. See _deployment.jsx for the
// speed/accuracy schema.
//
// A decision is prefill-only: the answer is read from the logits at the end of
// the prompt, so TTFT is the whole request latency, there is no TPOT (and no
// derived interactivity), and throughput per GPU is prompt tokens/s per GPU
// (osl = 0). TTFT is the P50 of end-to-end /v1/systemone request time.
//
// GB300: one GPU, the cell's recipe (no extra flag, so the server resolves
// --disable-radix-cache, --chunked-prefill-size -1 and trtllm_mha itself),
// PR #42645, checkpoint revision 3b45dea. Same closed-loop client and request
// shapes as the v1 benchmarks file: a 256-token state with one 4-option choice
// question (382 tokens) and an 8,192-token state with the same question
// (8,224 tokens). Section 3 of the page has the full tables.
//
// Accuracy through /v1/systemone with the v1 page's conversion: Belebele
// eng_Latn test (900) and WinoGrande xl validation (1,267). The checkpoint's
// reference DecisionModel scores 97.67 / 92.50 on the same items.
export const benchmarks = [
{
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
sglang_version: "PR #42645",
speed: [
{ workload: { dataset: "random", isl: 382, osl: 0, max_concurrency: 1, num_prompts: 64 },
ttft_ms: 52.6, tpot_ms: null, tokens_per_sec_per_gpu: 7203 },
{ workload: { dataset: "random", isl: 382, osl: 0, max_concurrency: 64, num_prompts: 256 },
ttft_ms: 931.4, tpot_ms: null, tokens_per_sec_per_gpu: 25770 },
{ workload: { dataset: "random", isl: 8224, osl: 0, max_concurrency: 1, num_prompts: 64 },
ttft_ms: 333.7, tpot_ms: null, tokens_per_sec_per_gpu: 24620 },
{ workload: { dataset: "random", isl: 8224, osl: 0, max_concurrency: 16, num_prompts: 64 },
ttft_ms: 5100.9, tpot_ms: null, tokens_per_sec_per_gpu: 25726 },
],
accuracy: { belebele_pct: 97.67, winogrande_pct: 92.42 },
notes: "Prefill-only model: TTFT is the full decision latency and throughput is prompt tokens/s/GPU. TPOT and interactivity do not apply.",
},
]
@@ -0,0 +1,138 @@
// Instantiated from cookbook-add-model/templates/config.jsx.tmpl.
// Single `export const config` literal with no spreads/calls/IIFE (Mintlify re-evals at hydration).
//
// PPLX-Decider-v1.1-27B: the v1 decision checkpoint's successor, same
// Qwen3.8-27B backbone and 255-row readout, trained with noncausal full
// attention (decision_config.json `attention_mode: noncausal_full_attention`).
// Every request is prefill-only, one pass per question with no decode, so there
// are no parser, speculative-decoding or PD-disaggregation knobs. BF16 is the
// only published precision.
//
// SGLang reads the attention mode and turns off the radix cache and chunked
// prefill itself, so no cell needs a flag for it. GB300: measured end to end
// on PR #42645 for speed, Belebele/WinoGrande accuracy, and per-item parity
// against the checkpoint's own reference DecisionModel. H200: same command,
// not run yet.
export const config = {
modelName: "PPLX-Decider-v1.1-27B",
latencyPercentile: "P50",
supportedHardware: ["h200", "gb300"],
variants: [
{ id: "default", label: "Default", subtitle: "27B dense, 1 GPU" },
],
quantizations: [{ id: "bf16", label: "BF16" }],
strategies: [{ id: "balanced", label: "Balanced" }],
nodesOptions: [{ id: "single", label: "Single Node" }],
modelNames: {
"default|bf16": "perplexity-ai/pplx-decider-v1.1-27b",
},
placeholders: {
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
PORT: { target: "command", label: "Bind port", default: "30000" },
HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
},
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/systemone \\
-H 'Content-Type: application/json' \\
-d '{"model":"{{MODEL_NAME}}","state":"My Stripe integration keeps failing. Please help ASAP.","questions":{"urgency":{"type":"noul","instructions":"Does this message express urgency?"}}}'`,
// Same sets and conversion as the v1 page, so the two pages compare directly.
accuracyLabels: [
["belebele_pct", "Belebele (eng_Latn, 900)", "%"],
["winogrande_pct", "WinoGrande (xl dev, 1267)", "%"],
],
// `lmsysorg/sglang:dev` is multi-arch (amd64 + arm64), so GB300 hosts pull
// the same tag as H200 hosts.
dockerImages: {
h200: "lmsysorg/sglang:dev",
gb300: "lmsysorg/sglang:dev",
},
github: { cookbookModel: "perplexity-ai/pplx-decider-v1.1-27b" },
// Every request is prefill-only (one pass per question, no decode), so the
// general axes that act on generation are omitted: parsers (no text output),
// speculative decoding and PD disaggregation (no decode phase). moe: dense
// model. CP / DP-Attention: no model-side support for this architecture.
// Prefix cache and hicache: noncausal attention rules out any prefix reuse,
// and the server turns the radix cache off itself.
playgroundFeatures: {
attention: {
knobs: [
{ id: "tp", label: "TP", values: [null, 1, 2, 4] },
],
},
flagSelects: [
{
// Replicas of the 1-GPU model behind one endpoint, the throughput lever.
id: "dp", title: "Data Parallel Replicas",
stripPrefixes: ["--dp-size"],
options: [
{ id: "1", label: "1" },
{ id: "2", label: "2", flags: ["--dp-size 2"] },
{ id: "4", label: "4", flags: ["--dp-size 4"] },
],
},
{
// All three run the full-attention layers noncausally. Auto resolves
// to trtllm_mha with 64-token pages on GB300.
id: "attnBackend", title: "Attention Backend",
stripPrefixes: ["--attention-backend", "--page-size"],
options: [
{ id: "auto", label: "Auto" },
{ id: "trtllm", label: "TRT-LLM MHA",
flags: ["--attention-backend trtllm_mha", "--page-size 64"],
disable: { hw: ["h200"] },
disableReason: "trtllm_mha is a Blackwell (SM100) kernel" },
{ id: "flashinfer", label: "FlashInfer", flags: ["--attention-backend flashinfer"] },
{ id: "triton", label: "Triton", flags: ["--attention-backend triton"] },
],
},
{
id: "gdnPrefill", title: "Gated DeltaNet Prefill Kernel",
stripPrefixes: ["--linear-attn-prefill-backend"],
options: [
{ id: "auto", label: "Auto" },
{ id: "triton", label: "Triton", flags: ["--linear-attn-prefill-backend triton"] },
{ id: "flashinfer", label: "FlashInfer", flags: ["--linear-attn-prefill-backend flashinfer"] },
],
},
],
},
cells: [
{
// Same command as GB300, not run on H200 yet.
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
// GB300 (SM103), one GPU. The server resolves --disable-radix-cache,
// --chunked-prefill-size -1 and trtllm_mha with 64-token pages on its
// own. Full attention runs noncausally through trtllm_mha's
// causal=False context kernel.
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
verified: true,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
],
}
@@ -143,6 +143,7 @@ def _configure_rocm_fp8_wo_a_gemm(model_config: Any, download_dir: str | None) -
def handle_model_specific_adjustments(server_args: Any):
cfg = resolving_view(server_args)
from sglang.srt.configs.model_config import (
NONCAUSAL_FULL_ATTENTION,
get_mimo_v2_fused_qkv_expected_tp_size,
is_deepseek_dsa,
)
@@ -205,6 +206,23 @@ def handle_model_specific_adjustments(server_args: Any):
f"{sorted(CP_DECODE_ATTN_TP_SUPPORTED_ARCHS)}."
)
decision_config = model_config.decision_config
if (
decision_config is not None
and decision_config.get("attention_mode") == NONCAUSAL_FULL_ATTENTION
):
# A cached prefix or a prefill chunk would attend to only part of its prompt.
logger.info(
"Radix cache and chunked prefill are disabled for a decision "
"checkpoint with noncausal full attention."
)
declare_resolution(
server_args,
"_handle_model_specific_adjustments",
disable_radix_cache=True,
chunked_prefill_size=-1,
)
_hybrid_spec = get_linear_attn_spec(hf_config)
if _hybrid_spec is not None and _hybrid_spec.uses_mamba_radix_cache:
handle_mamba_radix_cache(server_args, hf_config)
+21
View File
@@ -255,6 +255,10 @@ def is_dspark_draft(config) -> bool:
return _hf_arch(config) == "DSparkDraftModel"
# The decision_config.json attention mode whose full-attention layers read the whole prompt.
NONCAUSAL_FULL_ATTENTION = "noncausal_full_attention"
@lru_cache
def load_decision_config(model_path: str, revision: Optional[str]) -> Optional[dict]:
"""decision_config.json of a checkpoint whose LM head is a decision readout."""
@@ -273,6 +277,17 @@ def load_decision_config(model_path: str, revision: Optional[str]) -> Optional[d
f"decision_config.json format_version {config['format_version']} "
"is not supported, only version 1 is"
)
# Serving a mode this loader does not honor would silently change the answers.
attention_mode = config.get("attention_mode", "causal")
if attention_mode not in ("causal", NONCAUSAL_FULL_ATTENTION):
raise ValueError(
f"decision_config.json attention_mode {attention_mode!r} is not supported"
)
pooling = config.get("pooling", "last")
if pooling != "last":
raise ValueError(
f"decision_config.json pooling {pooling!r} is not supported, only 'last' is"
)
return config
@@ -523,6 +538,12 @@ class ModelConfig:
if self.decision_config is not None:
# A bare backbone whose readout the loader places in the LM head.
self.hf_config.architectures = ["Qwen3_5ForConditionalGeneration"]
if (
self.decision_config.get("attention_mode")
== NONCAUSAL_FULL_ATTENTION
):
# Full attention sees the whole prompt, while linear attention stays causal.
self.hf_text_config.is_causal = False
self.requires_mm_token_modalities = requires_mm_token_modalities(
self.hf_config.architectures, self.hf_text_config
)
@@ -1749,7 +1749,11 @@ class TRTLLMHAAttnBackend(FlashInferAttnBackend):
cum_seq_lens_kv=self.forward_metadata.cu_seqlens_k,
block_tables=page_table,
out_dtype=self.q_data_type,
mask_mode="causal",
mask_mode=(
"padding"
if layer.attn_type == AttentionType.ENCODER_ONLY
else "causal"
),
window_left=layer.sliding_window_size,
sinks=attention_sink,
skip_softmax_threshold_scale_factor=envs.SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR.get()
@@ -1794,6 +1798,7 @@ class TRTLLMHAAttnBackend(FlashInferAttnBackend):
out=out,
out_dtype=(None if uses_native_fp4 else self.q_data_type),
kv_cache_sf=kv_cache_block_scales,
causal=layer.attn_type != AttentionType.ENCODER_ONLY,
)
if cp_active:
+7 -1
View File
@@ -92,7 +92,7 @@ from sglang.srt.layers.quantization.unquant import (
UnquantizedLinearMethod,
bf16_gemm_dispatch,
)
from sglang.srt.layers.radix_attention import RadixAttention
from sglang.srt.layers.radix_attention import AttentionType, RadixAttention
from sglang.srt.layers.radix_linear_attention import RadixLinearAttention
from sglang.srt.layers.rotary_embedding import get_rope
from sglang.srt.layers.utils import PPMissingLayer, get_layer_id
@@ -1262,6 +1262,12 @@ class Qwen3_5AttentionDecoderLayer(nn.Module):
self.scaling,
num_kv_heads=self.num_kv_heads,
layer_id=layer_id,
# Noncausal decision checkpoints let full attention see the whole prompt.
attn_type=(
AttentionType.DECODER
if getattr(config, "is_causal", True)
else AttentionType.ENCODER_ONLY
),
prefix=f"{prefix}.attn",
quant_config=quant_config,
)
@@ -1,11 +1,13 @@
"""Unit tests for model configuration."""
import json
import os
import unittest
from tempfile import TemporaryDirectory
from types import SimpleNamespace
from unittest import mock
from transformers import LlamaConfig
from transformers import LlamaConfig, Qwen3_5Config
from sglang.srt.arg_groups.overrides import model_config_of
from sglang.srt.configs.model_config import (
@@ -13,6 +15,7 @@ from sglang.srt.configs.model_config import (
get_hybrid_layer_ids,
is_embedding_gemma,
is_multimodal_model,
load_decision_config,
register_model_config_factory,
resolve_spec_hidden_size,
)
@@ -213,5 +216,38 @@ class TestExternalModelConfig(CustomTestCase):
self.assertTrue(external.is_multimodal_breakable_cuda_graph_supported)
class TestDecisionCheckpointConfig(CustomTestCase):
def _checkpoint(self, directory, **decision):
Qwen3_5Config(architectures=["Qwen3_5Model"]).save_pretrained(directory)
decision = {"format_version": 1, "codes": ["A"], "token_ids": [32], **decision}
with open(os.path.join(directory, "decision_config.json"), "w") as f:
json.dump(decision, f)
def test_attention_mode_reaches_the_text_config(self):
"""A noncausal decision checkpoint must not be served with causal full attention."""
for decision, causal in (
({}, True),
({"attention_mode": "noncausal_full_attention"}, False),
):
with self.subTest(decision=decision), TemporaryDirectory() as checkpoint:
self._checkpoint(checkpoint, **decision)
config = ModelConfig(model_path=checkpoint)
self.assertEqual(
config.hf_config.architectures, ["Qwen3_5ForConditionalGeneration"]
)
if causal:
self.assertFalse(hasattr(config.hf_text_config, "is_causal"))
else:
self.assertIs(config.hf_text_config.is_causal, False)
def test_unsupported_decision_fields_are_refused(self):
"""An attention mode or pooling the server does not honor must not be served."""
for decision in ({"attention_mode": "bidirectional"}, {"pooling": "mean"}):
with self.subTest(decision=decision), TemporaryDirectory() as checkpoint:
self._checkpoint(checkpoint, **decision)
with self.assertRaisesRegex(ValueError, "is not supported"):
load_decision_config(checkpoint, None)
if __name__ == "__main__":
unittest.main()