[Feature] Add Aaronson-Gumbel text watermarking (#37577)

Co-authored-by: Nihar Patel <nihpat95@gmail.com>
This commit is contained in:
Xinyuan Tong
2026-10-13 12:47:50 -07:00
committed by GitHub
co-authored by Nihar Patel
parent 5eda6b7e67
commit 7dc2930585
46 changed files with 5569 additions and 52 deletions
+1
View File
@@ -932,6 +932,7 @@
"docs/advanced_features/session_radix_cache",
"docs/advanced_features/radix_eviction_policy",
"docs/advanced_features/hyperparameter_tuning",
"docs/advanced_features/text_watermarking",
"docs/advanced_features/attention_backend",
"docs/advanced_features/hisparse_guide",
"docs/advanced_features/speculative_decoding",
@@ -2707,6 +2707,18 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-watermark`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable Aaronson-Gumbel text watermarking.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--watermark-config`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Watermark settings as a JSON file path or an inline JSON object with optional key, key_b, context_window, mixing_probability, max_probability, default_enabled, and enforce_all fields.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: Optional[str]</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--flashinfer-mla-disable-ragged`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Not using ragged prefill wrapper when running flashinfer mla</td>
@@ -0,0 +1,391 @@
---
title: "Text Watermarking"
metatags:
description: "Configure Aaronson-Gumbel text watermarking and detect watermarked text."
---
SGLang provides opt-in Aaronson-Gumbel watermarking for sampled text. The watermark replaces random token selection with a keyed, deterministic choice that follows the same output distribution under the ideal independent-uniform hash model, so anyone holding the secret key can later test whether a token sequence carries the watermark. Detection needs only the key and the token IDs; it does not need the model or a GPU.
This feature can support transparency controls such as the machine-readable marking and detectability described in [Article 50 of the EU AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj). It is one technical signal, not a complete compliance program or proof of authorship.
## How it works
### Generation
At each sampling step, SGLang first applies frequency, presence, and repetition penalties, logit bias, and any structured-output grammar mask, followed by the request's temperature and its top-k, top-p, and min-p truncation, then renormalizes. Let `p_i` be the resulting probability of candidate token `i`; grammar-masked and truncated tokens have `p_i = 0` and can never be selected. For each candidate, SGLang hashes the key, the preceding token context, and the token ID into a uniform value `u_i` in `(0, 1)` and selects
```text
argmax over i with p_i > 0 of log(u_i) / p_i
```
When the `u_i` are independent uniforms, this rule selects token `i` with probability exactly `p_i`, so a single step is distributed like ordinary sampling. Ties go to the lowest token ID.
A position is watermarked only when all of the following hold; otherwise the request samples that position normally:
- The request has watermarking enabled (see [Per-request control](#per-request-control)).
- The request is not greedy. `temperature=0` is normalized to `top_k=1`, and `top_k=1` positions are never watermarked.
- The context is not empty.
- The context has not already been watermarked earlier in the same request. Runtime identifies a context by its 32-bit context hash.
- The request has watermarked fewer than `min(4,096, per-request token-pool capacity)` contexts.
- The largest truncated probability is at most the server's `max_probability`. The default `1.0` disables this gate.
Only a watermarked position enters the request's context history. A position skipped by the probability gate therefore leaves its context eligible for a later, higher-entropy occurrence, and once a request reaches its context budget the remaining tokens use ordinary sampling. The repeated-context rule keeps a repeated context from forcing the same token indefinitely.
The context for a position is the last `h` tokens before it, where `h` is the request's context window, taken from the prompt followed by the tokens generated so far. The first generated token therefore uses the last `h` prompt tokens. A context is shorter than `h` only when fewer than `h` tokens precede the position.
### Hash contract
All arithmetic wraps to unsigned 32 bits. The primitives are the MurmurHash3 x86 32-bit body mix and finalizer:
```text
mix(state, value):
value = value * 0xcc9e2d51
value = rotl32(value, 15)
value = value * 0x1b873593
state = rotl32(state XOR value, 13)
return state * 5 + 0xe6546b64
fmix32(value):
value = value XOR (value >> 16)
value = value * 0x85ebca6b
value = value XOR (value >> 13)
value = value * 0xc2b2ae35
return value XOR (value >> 16)
```
A key is 1 to 16 hexadecimal digits with an optional `0x` prefix, interpreted as an unsigned 64-bit integer; `key_lo` and `key_hi` are its low and high 32-bit words. For a context of `L` token IDs `c_1 ... c_L` and a candidate token `t`:
```text
context_hash = fmix32(mix(...mix(mix(0, c_1), c_2)..., c_L) XOR (4 * L))
token_hash = fmix32(mix(mix(mix(mix(0, key_lo), key_hi), context_hash), t) XOR 16)
u = (token_hash + 0.5) / 2^32
```
The context hash covers only the context tokens; keys enter only through the token hash and, in dual-key mode, through a per-context coin that chooses which key produces the token hash:
```text
coin = fmix32(mix(mix(mix(mix(mix(0, a_lo), a_hi), b_lo), b_hi), context_hash) XOR 20)
threshold = floor(P * 2^32)
key = key A if coin < threshold else key B
```
`a_lo`/`a_hi` and `b_lo`/`b_hi` are the words of key A and key B, and `P` is the server's `mixing_probability`.
### Detection
A detector holding the key scores each generated token `t` with `s = -log(1 - u)`, where `u` comes from the hash contract above using that token's context. Without the watermark, each `s` is an independent `Exponential(1)` sample. With the watermark, the generator preferred tokens with large `u`, so the scores are inflated.
Over `n` scored contexts, the score `S = sum(s)` follows `Gamma(shape=n, scale=1)` under the null hypothesis that the text is not watermarked with this key. The detector reports the upper-tail probability `P[Gamma(n, 1) >= S]` as the p-value, its natural logarithm (finite even when the p-value underflows to `0.0`), and the normal approximation `z = (S - n) / sqrt(n)`.
To match generation, a detector must follow these rules:
1. Use the exact prompt token IDs that preceded the completion and the request's context window `h`. With prompt IDs, the first completion token's context is the last `h` prompt tokens. Without prompt IDs, the first `h` completion tokens have unknown contexts and are skipped.
2. Score each distinct context once. The detector compares exact context token tuples, whereas generation compares 32-bit context hashes. After a hash collision, generation skips a later distinct context that the detector still scores. That extra score is an ordinary null sample, so the mismatch can only weaken detection, never create evidence.
3. Stop after the generation request's context budget and report a prefix test. The default detector budget is 4,096 distinct contexts. If the server's per-request token-pool capacity is lower, pass that lower value as `max_contexts`; scoring the unforced tail adds null samples that dilute the statistic without adding evidence. With a maximum-probability gate below `1.0`, the model-free detector cannot identify skipped positions, so this fixed prefix is conservative and may omit signal from later forced positions.
Positions that generation left unforced (greedy positions, and positions skipped by `max_probability`) are indistinguishable from ordinary text to the detector, which scores them as null samples; they dilute the statistic but cannot produce false evidence.
A low p-value is evidence that the text carries the keyed watermark. It is not a calibrated probability that a particular author or model produced the text. Applications must choose a decision threshold and measure false-positive and false-negative rates on their own traffic.
## Server configuration
`--enable-watermark` turns the feature on; every other watermark setting lives in one JSON object passed with `--watermark-config`, either as a file path or inline. Restrict the file to the server account and launch a CUDA server:
```json
{
"key": "0123456789abcdef",
"key_b": "fedcba9876543210",
"context_window": 4,
"mixing_probability": 0.5,
"max_probability": 1.0,
"default_enabled": true,
"enforce_all": false
}
```
```bash
chmod 600 /run/secrets/sglang-watermark.json
python -m sglang.launch_server \
--model-path Qwen/Qwen3-8B \
--enable-watermark \
--watermark-config /run/secrets/sglang-watermark.json
```
| Field | Default | Behavior |
| --- | --- | --- |
| `key` | unset | Server key (key A), 1 to 16 hex digits, for requests that enable watermarking without their own key. |
| `key_b` | unset | Enables dual-key mode with a second server key. Requires `key`. |
| `context_window` | `4` | Default context window and the maximum that a request may select, from 1 to 64 tokens. |
| `mixing_probability` | `0.5` | In dual-key mode, the probability of selecting key A for each context. Must be strictly between 0 and 1; a non-default value requires `key_b`. |
| `max_probability` | `1.0` | Skips watermarking when the largest truncated probability exceeds this value. Must be greater than 0 and at most 1; `1.0` disables the gate. |
| `default_enabled` | `false` | Watermarks requests that omit the `watermark` field with the server key. Requests may opt out. Requires `key`. |
| `enforce_all` | `false` | Watermarks requests that omit the `watermark` field and rejects `{"enabled": false}`. Requires `key`. |
Every field is optional and any other field is rejected. The config must be a JSON object of at most 4,096 bytes. A value that starts with `{` is parsed as inline JSON; any other value is read as a path to a regular file, and SGLang logs a warning when that file is readable by group or other users. Inline JSON is visible in the process command line, including `/proc/*/cmdline` on Linux, so pass keys through a file. `--watermark-config` requires `--enable-watermark`. A server key alone does not watermark every request; without `default_enabled` or `enforce_all`, requests must opt in.
SGLang redacts the keys and the `--watermark-config` value from the recorded launch command, startup logs, request logs, server information responses, and request crash-dump payloads. Process and GPU core dumps may contain in-memory keys. Protect those artifacts, the configuration file, and the transport used by clients that send per-request keys.
## Per-request control
The native `/generate` API and the OpenAI-compatible `/v1/completions` and `/v1/chat/completions` APIs accept a `watermark` object with three optional fields:
| Field | Type | Meaning |
| --- | --- | --- |
| `enabled` | boolean | `true` enables watermarking with the server key; `false` opts out. |
| `key` | hex string | Watermarks with this key instead of the server key. A key implicitly enables watermarking. |
| `context_window` | integer | Context window for this request, from 1 to the server's `context_window`. |
On `/generate`, pass `watermark` either at the top level of the request or inside `sampling_params`, but not both; a batched request may pass one object per prompt as a list. On the OpenAI-compatible APIs, pass it as a top-level request field:
```bash
curl http://127.0.0.1:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen/Qwen3-8B",
"messages": [{"role": "user", "content": "Explain photosynthesis."}],
"temperature": 1.0,
"top_p": 0.95,
"return_token_ids": true,
"watermark": {"enabled": true, "context_window": 4}
}'
```
The server resolves each request as follows:
| Server mode | No `watermark` field | `{"enabled": false}` | `{"enabled": true}` | `{"key": "HEX"}` |
| --- | --- | --- | --- | --- |
| Watermarking disabled | Off | Off | HTTP 400 | HTTP 400 |
| `--enable-watermark`, no server key | Off | Off | HTTP 400 | Request key |
| `--enable-watermark` with a server key | Off | Off | Server key | Request key |
| Add `default_enabled` | Server key | Off | Server key | Request key |
| Add `enforce_all` | Server key | HTTP 400 | Server key | Request key |
The request also returns HTTP 400 when:
- the `watermark` object sets neither `enabled` nor `key`, or contains an unknown field;
- `key` is combined with `"enabled": false`;
- a key is not 1 to 16 hexadecimal digits;
- `context_window` is not a positive integer or exceeds the server maximum;
- a watermarked request uses beam search.
Request-level keys are credentials carried in the request body. Use an authenticated TLS endpoint whenever they cross a host boundary, and avoid recording request bodies in proxies or application logs.
## Dual-key mode
`key_b` in `--watermark-config` applies dual-key selection to every watermarked request. The request's resolved key, whether the server key or a request key, is key A, and the server's second key is key B. A request key therefore replaces key A while key B stays active. Each eligible context uses key A with probability `mixing_probability` and key B otherwise, according to the coin in the [hash contract](#hash-contract); truncation, repeated-context, and greedy rules are unchanged. Without key B, selection uses the single-key path.
A detector holding both keys splits the distinct contexts by the coin, scores each context with its selected key, and tests the union, which is the strongest statistic; it can also test each partition separately. A detector holding only one key can score every distinct context with that key, but the contexts selected for the other key contribute only null samples and dilute the signal.
## Using the detector
`WatermarkDetector` implements the [detection rules](#detection) above. Construct it with the same key, key B, mixing probability, and context window that generation used:
A minimal [reference detection server](https://github.com/sgl-project/sglang/tree/main/examples/watermark) exposes the in-tree detector over HTTP and reads keys only from `--watermark-config`.
```python
WatermarkDetector(
key, # hex key A
*,
key_b=None, # hex key B for dual-key mode
mixing_probability=0.5, # must match the server's mixing_probability
context_window=4, # the request's effective window, 1 to 64
max_contexts=4096, # prefix-test length, 1 to 4096
)
```
- `detect_tokens(token_ids, *, prompt_token_ids=())` scores completion token IDs.
- `detect_text(text, tokenizer, *, prompt_token_ids=())` first encodes the text with `tokenizer.encode(text, add_special_tokens=False)`. Any object with that method works, including Hugging Face tokenizers.
- `detect(token_ids, *, key, key_b=None, prompt_token_ids=(), mixing_probability=0.5, context_window=4)` is a one-call shortcut.
Token IDs must be unsigned 32-bit integers; any iterable of Python or NumPy integers is accepted.
Every call returns a `WatermarkDetection`:
| Field | Meaning |
| --- | --- |
| `combined` | The test to report. Single-key: every distinct context scored with key A. Dual-key: each context scored with its coin-selected key. |
| `key_a_all_positions` | Every distinct context scored with key A. |
| `key_b_all_positions`, `key_a_partition`, `key_b_partition` | Dual-key only (`None` otherwise): every context scored with key B, and each coin partition scored with its own key. |
| `prefix_limit_reached` | `True` when scoring stopped at `max_contexts` distinct contexts. |
| `tokens_examined` | Completion tokens covered by the test, including repeated and skipped ones. |
| `repeated_contexts` | Positions skipped because their exact context was already scored. |
| `skipped_initial_tokens` | Leading completion tokens skipped because no prompt IDs were given. |
Each statistic is a `WatermarkStatistics` with `num_contexts`, `score`, `z_score`, `p_value`, and `log_p_value`.
Prefer the server's token IDs to re-tokenized text, because re-encoding decoded text does not always reproduce the generated IDs. With `"return_token_ids": true`, non-streaming chat completions return `prompt_token_ids` and `response_token_ids` on each choice, and completions return `prompt_token_ids` and `token_ids`.
### In an SGLang environment
```python
import requests
from sglang.srt.sampling.watermarking import WatermarkDetector
response = requests.post(
"http://127.0.0.1:30000/v1/chat/completions",
json={
"model": "Qwen/Qwen3-8B",
"messages": [{"role": "user", "content": "Explain photosynthesis."}],
"temperature": 1.0,
"top_p": 0.95,
"return_token_ids": True,
"watermark": {"enabled": True},
},
).json()
choice = response["choices"][0]
detector = WatermarkDetector("0123456789abcdef", context_window=4)
result = detector.detect_tokens(
choice["response_token_ids"],
prompt_token_ids=choice["prompt_token_ids"],
)
print(result.combined.num_contexts, result.combined.z_score, result.combined.log_p_value)
```
To score text, pass the model's tokenizer and, when available, the prompt IDs:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
result = detector.detect_text(
choice["message"]["content"],
tokenizer,
prompt_token_ids=choice["prompt_token_ids"],
)
```
For a dual-key server, pass both keys and the server's mixing probability:
```python
detector = WatermarkDetector(
"0123456789abcdef",
key_b="fedcba9876543210",
mixing_probability=0.5,
context_window=4,
)
result = detector.detect_tokens(completion_ids, prompt_token_ids=prompt_ids)
print(result.combined.p_value, result.key_a_partition.p_value, result.key_b_partition.p_value)
```
### Standalone, without SGLang or PyTorch
`detector.py` and `config.py` import only the Python standard library and `msgspec`, and the package's `__init__.py` imports only the detector. NumPy arrays are accepted as token-ID iterables, but NumPy is not required. To run detection in a separate application, copy the package directory and install `msgspec`:
```bash
mkdir -p detector_app
cp -R python/sglang/srt/sampling/watermarking detector_app/watermarking
python -m pip install msgspec
```
The copy also contains `core.py`, the server runtime, which imports PyTorch; nothing on the detector import path loads it. Run the application with `detector_app` on the import path and import the detector from the copied package:
```python
from watermarking.detector import WatermarkDetector
single = WatermarkDetector("0123456789abcdef", context_window=4)
result = single.detect_tokens(completion_ids, prompt_token_ids=prompt_ids)
dual = WatermarkDetector("0123456789abcdef", key_b="fedcba9876543210")
result = dual.detect_tokens(completion_ids, prompt_token_ids=prompt_ids)
```
Text detection needs only an object with `encode(text, *, add_special_tokens)` that returns a list of IDs. For example, with the lightweight `tokenizers` package:
```python
from tokenizers import Tokenizer
from watermarking.detector import WatermarkDetector
class TokenizerAdapter:
def __init__(self, name):
self._tokenizer = Tokenizer.from_pretrained(name)
def encode(self, text, *, add_special_tokens):
return self._tokenizer.encode(text, add_special_tokens=add_special_tokens).ids
detector = WatermarkDetector("0123456789abcdef", context_window=4)
result = detector.detect_text(text, TokenizerAdapter("Qwen/Qwen3-8B"), prompt_token_ids=prompt_ids)
```
A detector written from scratch instead of reusing this package must implement the [hash contract](#hash-contract) and all three [detection rules](#detection): prompt alignment, exact-tuple deduplication, and the 4,096-context prefix.
## Compatibility
| Feature | Status | Behavior and validation |
| --- | --- | --- |
| CUDA graphs | Verified | Decode graphs, eager fallback, and fused state updates use the same selector contract. |
| Overlap scheduling | Verified | Prompt initialization is carried with the delayed sampling batch. |
| Tensor parallelism | Verified | TP2 and TP4 ordinary decode, plus NEXTN with TP2, produce detectable output without rank divergence. |
| Mixed enabled and disabled requests | Verified | Keys, context windows, histories, and enablement are resolved per request. |
| JSON schema and regex grammars | Verified | Grammar masking runs before watermark selection; strict outputs remain valid and detectable. |
| Streaming chat completions | Verified | Streaming and non-streaming requests use the same watermark state and token selection. |
| Parallel sampling (`n > 1`) | Verified | Each generated choice owns independent request state and history. |
| Output log probabilities | Verified | Ordinary and speculative decode report the pre-watermark target distribution. Logits are cloned only for requests that ask for log probabilities. |
| Top-k, top-p, and min-p | Verified | The selector reproduces SGLang's combined truncation support before keyed selection. |
| Maximum-probability gate | Verified | Positions above `max_probability` use ordinary sampling and do not consume the repeated-context budget. |
| Logit bias and frequency, presence, and repetition penalties | Verified | These transforms run before watermark selection. |
| Chunked prefill | Verified | Each prefill chunk rebuilds the prompt tail from the request's complete logical fill IDs. |
| Radix-cache prompt reuse | Verified | Prompt-tail initialization uses logical request IDs rather than only newly computed tokens. |
| Decode retraction | Verified | Re-prefill restores the distinct-context history before generation resumes. |
| Request abort and request-slot reuse | Verified | A later prefill resets ring-buffer and context-history state for the reused slot. |
| Multimodal text generation | Verified | Qwen3-VL text tokens use the same watermark path after multimodal prefill. |
| DP attention | Verified | DP2/TP2 normal and NEXTN decoding were verified with Qwen3-8B on two H200 GPUs, including idle-rank batches. |
| NGRAM speculative decoding | Verified | Draft-node contexts are reconstructed before verification sampling. |
| EAGLE and EAGLE3 speculative decoding | Verified | Tree nodes use their own ancestor-derived contexts. |
| NEXTN speculative decoding | Verified | NEXTN resolves to the EAGLE verification path. |
| Pipeline-parallel speculative decoding | Rejected | Startup validation fails closed. |
| Other speculative algorithms | Rejected | Startup validation fails closed. |
| Speculative rejection sampling | Rejected | Startup validation fails closed. |
| Custom logit processors | Rejected | Startup validation fails closed because processor ordering is not defined. |
| Token-oracle sampling backend | Rejected | Startup validation fails closed because oracle-selected tokens bypass the watermark selector. |
| Beam search | Rejected | Request validation rejects watermarked beam-search requests. |
| Diffusion LLM | Rejected | Startup validation fails closed. |
| PD disaggregation | Rejected | Startup validation fails closed because watermark state is not transferred between workers. |
| Embedded Rust API server | Rejected | Startup validation fails closed. |
| Non-CUDA devices | Rejected | Startup validation fails closed. |
In speculative decoding, verification clones the target logits before forcing, so target-model log-probability computation still sees the pre-watermark distribution. Each draft node gets its own context, rebuilt from its ancestors in the verify tree. Repeated-context admission is conservative across the whole tree: once a context hash appears in an earlier row, later rows with the same hash are not forced, even on sibling branches. This keeps duplicate entries out of the history, but reconvergent wide trees may carry less signal.
## Limitations and performance
- Greedy requests carry no signal, and very low-entropy distributions leave few meaningful alternatives for the selector.
- A `max_probability` value below 1 keeps ordinary selection at low-entropy positions and can improve speculative draft acceptance, but the unforced positions dilute the detector statistic.
- Short completions provide few distinct contexts and therefore weak statistical power.
- Paraphrasing, rewriting, translation, and re-tokenization can weaken or remove the signal.
- Detection needs the matching key or keys, mixing probability, context window, and tokenizer, and is strongest with the exact prompt token IDs.
- A request watermarks at most `min(4,096, per-request token-pool capacity)` distinct contexts; later tokens use ordinary sampling.
- The hashes are MurmurHash3, a keyed but non-cryptographic mix. There is no indistinguishability or unforgeability guarantee, and anyone holding the key can both detect and forge the watermark.
- Speculative decoding can accept fewer draft tokens when a draft proposal does not match the watermarked target token. The impact depends on tree width and workload.
- Mixed batches choose the selector path from the batch-wide maximum top-k. A transient unlimited-top-k request can keep that batch on the full-sort path until its sampling metadata is rebuilt.
- Retraction reconstructs a conservative distinct-context history. Repeated retractions can reduce later signal and add scheduler-side replay work.
On one single-GPU node of each type with public `Qwen/Qwen3-8B`, TP1, TRT-LLM MHA, ordinary decoding, temperature 1.0, top-p 0.95, unlimited top-k, and 800-token sampled outputs, the fused selector measured the following mean output throughput over three repetitions per cell:
| Node | Concurrency | Watermark off | Watermark on | Change |
| --- | ---: | ---: | ---: | ---: |
| GB300 | 1 | 270.25 token/s | 260.44 token/s | -3.63% |
| GB300 | 8, per stream | 224.88 token/s | 212.84 token/s | -5.35% |
| B300 | 1 | 258.84 token/s | 247.69 token/s | -4.31% |
| B300 | 8, per stream | 196.26 token/s | 182.04 token/s | -7.24% |
The GB300 concurrency-eight row was remeasured after the score-precision fix on a default-off server by comparing omitted requests with request-level opt-in. These measurements describe one model, GPU, sampling configuration, and output length. Benchmark your deployment before setting capacity targets. Finite-top-k selector latency at batch size one is also hardware-dependent: in one microbenchmark shape (`top_k=20`), the fast path was 58.8% slower than a full sort on H200 with bfloat16 logits and 6.4% faster on GB300 with float32 logits, and at batch size eight it was faster on both.
On a 2x GB300 node with public `Qwen/Qwen3.5-9B`, TP2, NEXTN 3/1/4, concurrency 8, 1,024-token random inputs, 512-token outputs, temperature 0.8, top-p 0.95, and top-k 20, the fused speculative path measured the following means over three repetitions:
| Watermark | Mean TPOT | Aggregate output throughput | Mean accept length |
| --- | ---: | ---: | ---: |
| Off | 2.246 ms | 3,188.9 token/s | 2.759 |
| On | 2.451 ms | 2,922.2 token/s | 2.783 |
| Change | +9.17% | -8.36% | +0.85% |
Each speculative row still requires keyed selection, but tree-context reconstruction, repeated-context admission, accepted-context recording, and ring-buffer append run as fused GPU operations. The reported cost remains workload- and acceptance-dependent.
## See also
The fused Triton selector is adapted from Meta's Apache-2.0 [TextSeal](https://github.com/facebookresearch/textseal) implementation and re-implemented on the hash contract above, so detection follows this page rather than TextSeal's own tooling.
The runtime and reference detector live in `python/sglang/srt/sampling/watermarking/`. Registered coverage lives in `test/registered/kernels/ops/sampling/test_watermark_*.py` and `test/registered/e2e/openai_server/test_watermark.py`.
+6
View File
@@ -0,0 +1,6 @@
# Watermark detection server
The reference HTTP server wraps the in-tree detector; keys and the context window come only from the server's `--watermark-config` JSON (a file path or inline object), never request bodies.
Run: `python examples/watermark/detection_server.py --watermark-config /run/secrets/sglang-watermark.json --tokenizer Qwen/Qwen3-8B`.
Token IDs: `curl localhost:8000/detect -H 'Content-Type: application/json' -d '{"token_ids":[1,2,3,4,5]}'`.
Text requests use `{"text":"Text to inspect"}` and may set a `tokenizer` identifier when no default was provided; protect this reference endpoint because tokenizer loading and detector scores are externally observable.
+166
View File
@@ -0,0 +1,166 @@
"""Minimal reference server for watermark detection."""
import argparse
from functools import lru_cache
from typing import Optional
import uvicorn
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, ConfigDict, Field, model_validator
from transformers import AutoTokenizer, PreTrainedTokenizerBase
from sglang.srt.sampling.watermarking import (
WatermarkDetection,
WatermarkDetector,
WatermarkStatistics,
)
from sglang.srt.sampling.watermarking.config import load_watermark_config
class DetectionRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
text: Optional[str] = None
token_ids: Optional[list[int]] = None
prompt_token_ids: list[int] = Field(default_factory=list)
tokenizer: Optional[str] = None
@model_validator(mode="after")
def validate_input(self) -> "DetectionRequest":
if (self.text is None) == (self.token_ids is None):
raise ValueError("exactly one of text or token_ids is required")
return self
class DetectionStatistics(BaseModel):
p_value: float
z_score: float
num_contexts: int
class DetectionResponse(DetectionStatistics):
watermarked: bool
per_key: Optional[dict[str, DetectionStatistics]] = None
@lru_cache(maxsize=4)
def get_tokenizer(tokenizer_id: str) -> PreTrainedTokenizerBase:
return AutoTokenizer.from_pretrained(tokenizer_id, trust_remote_code=False)
def statistics_response(statistics: WatermarkStatistics) -> DetectionStatistics:
return DetectionStatistics(
p_value=statistics.p_value,
z_score=statistics.z_score,
num_contexts=statistics.num_contexts,
)
def per_key_response(
result: WatermarkDetection,
) -> Optional[dict[str, DetectionStatistics]]:
if result.key_b_all_positions is None:
return None
return {
"key_a_all_positions": statistics_response(result.key_a_all_positions),
"key_b_all_positions": statistics_response(result.key_b_all_positions),
"key_a_partition": statistics_response(result.key_a_partition),
"key_b_partition": statistics_response(result.key_b_partition),
}
def create_app(
detector: WatermarkDetector,
*,
p_value_threshold: float,
default_tokenizer: Optional[str],
) -> FastAPI:
app = FastAPI()
@app.get("/health")
def health() -> dict[str, str]:
return {"status": "ok"}
@app.post("/detect")
def detect(request: DetectionRequest) -> DetectionResponse:
if request.text is not None:
tokenizer_id = request.tokenizer or default_tokenizer
if tokenizer_id is None:
raise HTTPException(
status_code=400,
detail="text detection requires a tokenizer",
)
try:
token_ids = get_tokenizer(tokenizer_id).encode(
request.text, add_special_tokens=False
)
except (OSError, ValueError) as error:
raise HTTPException(
status_code=400, detail="failed to load tokenizer"
) from error
else:
token_ids = request.token_ids
try:
result = detector.detect_tokens(
token_ids, prompt_token_ids=request.prompt_token_ids
)
except ValueError as error:
raise HTTPException(status_code=400, detail=str(error)) from error
combined = statistics_response(result.combined)
return DetectionResponse(
**combined.model_dump(),
watermarked=combined.p_value < p_value_threshold,
per_key=per_key_response(result),
)
return app
def probability(value: str) -> float:
probability_value = float(value)
if not 0 < probability_value < 1:
raise argparse.ArgumentTypeError("value must be strictly between 0 and 1")
return probability_value
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--watermark-config", required=True)
parser.add_argument("--tokenizer")
parser.add_argument("--max-contexts", type=int, default=4096)
parser.add_argument("--p-value-threshold", type=probability, default=0.01)
parser.add_argument("--host", default="127.0.0.1")
parser.add_argument("--port", type=int, default=8000)
return parser.parse_args()
def main(args: argparse.Namespace) -> None:
config = load_watermark_config(args.watermark_config)
if config.key is None:
raise ValueError("watermark config must set key")
detector = WatermarkDetector(
config.key,
key_b=config.key_b,
mixing_probability=(
config.mixing_probability if config.mixing_probability is not None else 0.5
),
context_window=(
config.context_window if config.context_window is not None else 4
),
max_contexts=args.max_contexts,
)
uvicorn.run(
create_app(
detector,
p_value_threshold=args.p_value_threshold,
default_tokenizer=args.tokenizer,
),
host=args.host,
port=args.port,
)
if __name__ == "__main__":
main(parse_args())
+10 -1
View File
@@ -1,5 +1,5 @@
// The API face of SamplingParams — the ground truth for
// rust/sglang-server/src/message/sampling.rs's 25 API fields (its 5
// rust/sglang-server/src/message/sampling.rs's 27 API fields (its 6
// normalize-internal fields and the msgpack scheduler encoding are IPC, not
// API, and stay hand-written there). Unknown JSON keys are DENIED: parity with
// Python's `SamplingParams(**kwargs)` TypeError.
@@ -97,4 +97,13 @@ message SamplingParams {
// beam_width > 1 turns the request into a beam search request; `n` is then
// the number of returned sequences (n <= beam_width).
optional int64 beam_width = 26;
WatermarkRequestConfig watermark = 27;
}
message WatermarkRequestConfig {
option (sglang.json.v1.message_json).unknown_fields = UNKNOWN_FIELDS_DENY;
optional bool enabled = 1;
optional string key = 2;
optional int64 context_window = 3;
}
+52 -1
View File
@@ -203,6 +203,7 @@ class SamplingParams(msgspec.Struct, kw_only=True, frozen=True):
sampling_seed: Optional[int] = None
custom_params: Optional[Any] = None
beam_width: Optional[int] = None
watermark: Optional[WatermarkRequestConfig] = None
@classmethod
def from_json_value(cls, value: Any) -> "SamplingParams":
@@ -214,6 +215,23 @@ class SamplingParams(msgspec.Struct, kw_only=True, frozen=True):
return encode_SamplingParams(self)
class WatermarkRequestConfig(msgspec.Struct, kw_only=True, frozen=True):
"""sglang.api.v1.WatermarkRequestConfig: JSON object (unknown keys rejected)."""
enabled: Optional[bool] = None
key: Optional[str] = None
context_window: Optional[int] = None
@classmethod
def from_json_value(cls, value: Any) -> "WatermarkRequestConfig":
"""Decode a parsed JSON value; raises JsonContractError on a contract violation."""
return decode_WatermarkRequestConfig(value)
def to_json_value(self) -> Dict[str, Any]:
"""Encode to a JSON-ready value (json.dumps-compatible)."""
return encode_WatermarkRequestConfig(self)
class GenerateRequest(msgspec.Struct, kw_only=True, frozen=True):
"""sglang.api.v1.GenerateRequest: JSON object (unknown keys ignored)."""
@@ -877,7 +895,7 @@ def encode_FinishReason(v: FinishReason) -> Any:
raise TypeError(f"FinishReason: unsupported value {type(v).__name__}")
_FIELDS_SamplingParams = ("max_new_tokens", "stop", "stop_token_ids", "stop_regex", "temperature", "top_p", "top_k", "min_p", "frequency_penalty", "presence_penalty", "repetition_penalty", "min_new_tokens", "n", "json_schema", "regex", "ebnf", "structural_tag", "ignore_eos", "skip_special_tokens", "spaces_between_special_tokens", "no_stop_trim", "stream_interval", "logit_bias", "sampling_seed", "custom_params", "beam_width",)
_FIELDS_SamplingParams = ("max_new_tokens", "stop", "stop_token_ids", "stop_regex", "temperature", "top_p", "top_k", "min_p", "frequency_penalty", "presence_penalty", "repetition_penalty", "min_new_tokens", "n", "json_schema", "regex", "ebnf", "structural_tag", "ignore_eos", "skip_special_tokens", "spaces_between_special_tokens", "no_stop_trim", "stream_interval", "logit_bias", "sampling_seed", "custom_params", "beam_width", "watermark",)
def decode_SamplingParams(value: Any) -> SamplingParams:
@@ -936,6 +954,8 @@ def decode_SamplingParams(value: Any) -> SamplingParams:
kw["custom_params"] = x
elif key == "beam_width":
kw["beam_width"] = None if x is None else _expect_int(x, expected="i64")
elif key == "watermark":
kw["watermark"] = None if x is None else decode_WatermarkRequestConfig(x)
else:
raise _unknown_field(key, fields=_FIELDS_SamplingParams)
return SamplingParams(**kw)
@@ -994,6 +1014,37 @@ def encode_SamplingParams(v: SamplingParams) -> Dict[str, Any]:
d["custom_params"] = v.custom_params
if v.beam_width is not None:
d["beam_width"] = v.beam_width
if v.watermark is not None:
d["watermark"] = encode_WatermarkRequestConfig(v.watermark)
return d
_FIELDS_WatermarkRequestConfig = ("enabled", "key", "context_window",)
def decode_WatermarkRequestConfig(value: Any) -> WatermarkRequestConfig:
m = _expect_map(value, expected="struct WatermarkRequestConfig")
kw: Dict[str, Any] = {}
for key, x in m.items():
if key == "enabled":
kw["enabled"] = None if x is None else _expect_bool(x, expected="a boolean")
elif key == "key":
kw["key"] = None if x is None else _expect_str(x, expected="a string")
elif key == "context_window":
kw["context_window"] = None if x is None else _expect_int(x, expected="i64")
else:
raise _unknown_field(key, fields=_FIELDS_WatermarkRequestConfig)
return WatermarkRequestConfig(**kw)
def encode_WatermarkRequestConfig(v: WatermarkRequestConfig) -> Dict[str, Any]:
d: Dict[str, Any] = {}
if v.enabled is not None:
d["enabled"] = v.enabled
if v.key is not None:
d["key"] = v.key
if v.context_window is not None:
d["context_window"] = v.context_window
return d
File diff suppressed because it is too large Load Diff
+2 -1
View File
@@ -130,7 +130,8 @@ DISAGG_TRANSFER_BACKEND_CHOICES = [
GRAMMAR_BACKEND_CHOICES = ["xgrammar", "outlines", "llguidance", "none"]
SAMPLING_BACKEND_CHOICES = {"flashinfer", "pytorch", "ascend", "intel_xpu"}
BUILT_IN_SAMPLING_BACKENDS = frozenset({"flashinfer", "pytorch", "ascend", "intel_xpu"})
SAMPLING_BACKEND_CHOICES = set(BUILT_IN_SAMPLING_BACKENDS)
MOE_RUNNER_BACKEND_CHOICES = [
"auto",
@@ -69,6 +69,44 @@ class ExecFeatures(msgspec.Struct):
bool,
"Enable users to pass custom logit processors to the server (disabled by default for security)",
] = False
enable_watermark: A[
bool,
"Enable Aaronson-Gumbel text watermarking.",
] = False
watermark_config: A[
Optional[str],
"Watermark settings as a JSON file path or an inline JSON object with "
"optional key, key_b, context_window, mixing_probability, max_probability, "
"default_enabled, and enforce_all fields.",
] = None
watermark_key: A[
Optional[str],
Arg(help="Resolved from --watermark-config.", no_cli=True, resolvable=True),
] = None
watermark_key_b: A[
Optional[str],
Arg(help="Resolved from --watermark-config.", no_cli=True, resolvable=True),
] = None
watermark_context_window: A[
int,
Arg(help="Resolved from --watermark-config.", no_cli=True, resolvable=True),
] = 4
watermark_mixing_probability: A[
float,
Arg(help="Resolved from --watermark-config.", no_cli=True, resolvable=True),
] = 0.5
watermark_max_probability: A[
float,
Arg(help="Resolved from --watermark-config.", no_cli=True, resolvable=True),
] = 1.0
watermark_default_enabled: A[
bool,
Arg(help="Resolved from --watermark-config.", no_cli=True, resolvable=True),
] = False
watermark_enforce_all: A[
bool,
Arg(help="Resolved from --watermark-config.", no_cli=True, resolvable=True),
] = False
enable_return_hidden_states: A[
bool,
"Enable returning full hidden states with responses. Equivalent to "
+2
View File
@@ -63,10 +63,12 @@ def run_resolution_pipeline(server_args: Any) -> None:
handle_return_hidden_states_mode,
handle_ssl_validation,
handle_tokenizer_batching,
handle_watermark_config,
)
run_hook(handle_return_hidden_states_mode, server_args)
run_hook(handle_media_url_security, server_args)
run_hook(handle_watermark_config, server_args)
from sglang.srt.arg_groups.hicache_hook import (
handle_hicache,
handle_hicache_ratio_default,
@@ -17,6 +17,7 @@ _OVERRIDABLE_HOOKS: FrozenSet[str] = frozenset(
"handle_mega_moe",
"handle_return_hidden_states_mode",
"handle_media_url_security",
"handle_watermark_config",
"handle_hicache_ratio_default",
"handle_offload_compatibility",
"validate_prefill_decode_interval",
+25 -4
View File
@@ -18,6 +18,7 @@ from sglang.srt.arg_groups.overrides import (
from sglang.srt.environ import envs
from sglang.srt.model_executor.cuda_graph_config import Backend, Phase, with_phase
from sglang.srt.runtime_context import get_platform
from sglang.srt.sampling.watermarking.config import load_watermark_config
from sglang.srt.utils.common import (
configure_media_url_security,
get_device,
@@ -29,6 +30,20 @@ from sglang.utils import is_in_ci
logger = logging.getLogger(__name__)
def handle_watermark_config(server_args: Any) -> None:
cfg = resolving_view(server_args)
if cfg.watermark_config is None:
return
config = load_watermark_config(cfg.watermark_config)
resolved = {
f"watermark_{field}": getattr(config, field)
for field in config.__struct_fields__
if getattr(config, field) is not None
}
if resolved:
declare_resolution(server_args, "handle_watermark_config", **resolved)
def handle_ssl_validation(server_args: Any):
"""Ensure SSL arguments are consistent and referenced files exist."""
cfg = resolving_view(server_args)
@@ -570,19 +585,25 @@ def handle_other_validations(server_args: Any):
)
# Validate preferred_sampling_params
if cfg.preferred_sampling_params:
if isinstance(cfg.preferred_sampling_params, str):
preferred_sampling_params = cfg.preferred_sampling_params
if preferred_sampling_params:
if isinstance(preferred_sampling_params, str):
preferred_sampling_params = json.loads(preferred_sampling_params)
declare_resolution(
server_args,
"_handle_other_validations",
preferred_sampling_params=json.loads(cfg.preferred_sampling_params),
preferred_sampling_params=preferred_sampling_params,
)
if "watermark" in preferred_sampling_params:
raise ValueError(
"watermark is not supported in --preferred-sampling-params"
)
# Validate preferred_sampling_params doesn't use tokenizer-dependent features
if cfg.skip_tokenizer_init:
from sglang.srt.sampling.sampling_params import SamplingParams
test_params = SamplingParams(**cfg.preferred_sampling_params)
test_params = SamplingParams(**preferred_sampling_params)
# raises if tokenizer-dependent features used
test_params.normalize(None)
@@ -8,6 +8,7 @@ import logging
import os
from typing import Any, Dict, List, Optional
from sglang.srt.arg_groups.choices import BUILT_IN_SAMPLING_BACKENDS
from sglang.srt.arg_groups.overrides import (
_hisparse_validation,
resolved_view,
@@ -19,12 +20,91 @@ from sglang.srt.distributed.device_communicators.mooncake_transfer_engine import
)
from sglang.srt.environ import envs
from sglang.srt.runtime_context import attn_dp_enabled_of, get_platform, num_dp_ranks_of
from sglang.srt.sampling.watermarking.config import (
MAX_WATERMARK_CONTEXT_WINDOW,
parse_watermark_key,
)
from sglang.srt.utils.common import torch_release
from sglang.srt.utils.runai_utils import is_runai_obj_uri
logger = logging.getLogger(__name__)
def check_watermark_server_args(server_args: Any) -> None:
cfg = resolving_view(server_args)
if cfg.watermark_config is not None and not cfg.enable_watermark:
raise ValueError("--watermark-config requires --enable-watermark")
if not cfg.enable_watermark:
return
if cfg.device != "cuda":
raise ValueError(
f"--enable-watermark requires --device cuda, got {cfg.device!r}"
)
if cfg.watermark_key is not None:
parse_watermark_key(cfg.watermark_key)
if cfg.watermark_key_b is not None:
parse_watermark_key(cfg.watermark_key_b)
if (
isinstance(cfg.watermark_context_window, bool)
or not isinstance(cfg.watermark_context_window, int)
or not 1 <= cfg.watermark_context_window <= MAX_WATERMARK_CONTEXT_WINDOW
):
raise ValueError("watermark context_window must be an integer from 1 to 64")
if cfg.watermark_key_b is not None and cfg.watermark_key is None:
raise ValueError("watermark key_b requires a server key")
if not 0 < cfg.watermark_mixing_probability < 1:
raise ValueError(
"watermark mixing_probability must be strictly between 0 and 1"
)
if cfg.watermark_key_b is None and cfg.watermark_mixing_probability != 0.5:
raise ValueError("watermark mixing_probability requires key_b")
if not 0 < cfg.watermark_max_probability <= 1:
raise ValueError(
"watermark max_probability must be greater than 0 and at most 1"
)
if (
cfg.watermark_default_enabled or cfg.watermark_enforce_all
) and cfg.watermark_key is None:
raise ValueError(
"watermark default_enabled and enforce_all require a server key"
)
if cfg.enable_custom_logit_processor:
raise ValueError(
"--enable-watermark is incompatible with --enable-custom-logit-processor"
)
if (
cfg.sampling_backend is not None
and cfg.sampling_backend not in BUILT_IN_SAMPLING_BACKENDS
):
raise ValueError(
"--enable-watermark does not support custom sampling backends; "
f"got --sampling-backend {cfg.sampling_backend!r}"
)
if cfg.dllm_algorithm is not None:
raise ValueError("--enable-watermark is not supported with diffusion LLM")
if cfg.disaggregation_mode != "null":
raise ValueError("--enable-watermark is not supported with PD disaggregation")
if cfg.speculative_algorithm not in {None, "NGRAM", "EAGLE", "EAGLE3"}:
raise ValueError(
"--enable-watermark supports speculative algorithms NGRAM, EAGLE, "
f"and EAGLE3, got {cfg.speculative_algorithm!r}"
)
if cfg.pp_size > 1 and cfg.speculative_algorithm is not None:
raise ValueError(
"--enable-watermark is not supported with pipeline-parallel "
"speculative decoding"
)
if cfg.speculative_use_rejection_sampling:
raise ValueError(
"--enable-watermark is incompatible with "
"--speculative-use-rejection-sampling"
)
if envs.SGLANG_RUST_SERVER.get():
raise ValueError("--enable-watermark is not supported with SGLANG_RUST_SERVER")
def validate_response_store(server_args: Any) -> None:
cfg = resolving_view(server_args)
if cfg.enable_response_store and cfg.disaggregation_mode != "null":
@@ -295,6 +375,8 @@ def check_server_args(server_args: Any):
"--kv-canary-sweep-interval requires --kv-canary in {log, raise}"
)
check_watermark_server_args(server_args)
check_load_publish_args(server_args)
+22 -11
View File
@@ -125,6 +125,7 @@ from sglang.srt.runtime_context import (
restore_context,
snapshot_context,
)
from sglang.srt.sampling.watermarking.core import redact_watermark_secrets
from sglang.srt.server_args import PortArgs, ServerArgs
from sglang.srt.utils import (
MultiprocessingSerializer,
@@ -166,6 +167,10 @@ asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
_is_cuda = is_cuda()
def _server_args_for_logging(server_args: ServerArgs) -> Dict[str, Any]:
return redact_watermark_secrets(server_args.resolved_dict())
@dataclasses.dataclass
class SchedulerInitResult:
"""Result from launching schedulers."""
@@ -291,10 +296,14 @@ class Engine(EngineScoreMixin, EngineBase):
msgspec.Struct.__setattr__(
server_args,
"_launch_command",
"Engine(" + ", ".join(f"{k}={v!r}" for k, v in kwargs.items()) + ")",
"Engine("
+ ", ".join(
f"{k}={v!r}" for k, v in redact_watermark_secrets(kwargs).items()
)
+ ")",
)
self.server_args = server_args
logger.info(f"server_args={server_args.resolved_dict()}")
logger.info(f"server_args={_server_args_for_logging(server_args)}")
# Rust Server is not supported with the offline Engine API
if envs.SGLANG_RUST_SERVER.get():
@@ -1167,7 +1176,7 @@ class Engine(EngineScoreMixin, EngineBase):
# Allocate ports for inter-process communications
if port_args is None:
port_args = PortArgs.init_new(server_args)
logger.info(f"server_args={server_args.resolved_dict()}")
logger.info(f"server_args={_server_args_for_logging(server_args)}")
# Start the engine info bootstrap server if per-rank info is needed.
engine_info_bootstrap_server = None
@@ -1551,14 +1560,16 @@ class Engine(EngineScoreMixin, EngineBase):
self.tokenizer_manager.get_internal_state()
)
return msgspec_to_builtins(
{
**self.tokenizer_manager.server_args.resolved_dict(),
"launch_command": self.tokenizer_manager.server_args.launch_command,
**self._scheduler_init_result.scheduler_infos[0],
"startup_time": self.tokenizer_manager.startup_time,
"internal_states": internal_states,
"version": __version__,
}
redact_watermark_secrets(
{
**self.tokenizer_manager.server_args.resolved_dict(),
"launch_command": self.tokenizer_manager.server_args.launch_command,
**self._scheduler_init_result.scheduler_infos[0],
"startup_time": self.tokenizer_manager.startup_time,
"internal_states": internal_states,
"version": __version__,
}
)
)
def get_model_info(self):
+5 -2
View File
@@ -17,6 +17,7 @@ from pydantic import ValidationError
from sglang.srt.arg_groups.overrides import resolving_view
from sglang.srt.configs.embedding_model_spec import resolved_embedding_plan
from sglang.srt.runtime_context import get_lora, get_serving
from sglang.srt.sampling.watermarking.core import redact_watermark_secrets
from sglang.srt.utils.common import build_server_info
from sglang.srt.utils.msgspec_utils import msgspec_to_builtins
@@ -427,8 +428,10 @@ class RuntimeHandle:
def get_server_info(self) -> str:
return json.dumps(
msgspec_to_builtins(
build_server_info(
self.tokenizer_manager.server_args, self.scheduler_info
redact_watermark_secrets(
build_server_info(
self.tokenizer_manager.server_args, self.scheduler_info
)
)
),
default=str,
+10 -7
View File
@@ -176,6 +176,7 @@ from sglang.srt.observability.trace import (
)
from sglang.srt.parser.reasoning_parser import ReasoningParser
from sglang.srt.parser.template_manager import TemplateManager
from sglang.srt.sampling.watermarking.core import redact_watermark_secrets
from sglang.srt.server_args import PortArgs, ServerArgs
from sglang.srt.utils import (
add_prometheus_middleware,
@@ -863,13 +864,15 @@ async def server_info():
server_args = _global_state.tokenizer_manager.server_args
return msgspec_to_builtins(
{
**build_server_info(server_args, _global_state.scheduler_info),
"startup_time": _global_state.tokenizer_manager.startup_time,
"internal_states": internal_states,
"version": __version__,
"frontend": "python",
}
redact_watermark_secrets(
{
**build_server_info(server_args, _global_state.scheduler_info),
"startup_time": _global_state.tokenizer_manager.startup_time,
"internal_states": internal_states,
"version": __version__,
"frontend": "python",
}
)
)
@@ -398,6 +398,13 @@ class CompletionRequest(PDRoutingFields):
response_format: Optional[Union[ResponseFormat, StructuralTagResponseFormat]] = None
custom_params: Optional[Dict] = None
custom_logit_processor: Optional[str] = None
watermark: Optional[Dict[str, Any]] = Field(
default=None,
description=(
"Per-request watermark settings: enabled, key, and context_window. "
"A key implicitly enables watermarking."
),
)
images_config: Optional[Dict] = None
@@ -958,6 +965,13 @@ class ChatCompletionRequest(PDRoutingFields):
# Custom logit processor for advanced sampling control
custom_logit_processor: Optional[Union[List[Optional[str]], str]] = None
custom_params: Optional[Dict] = None
watermark: Optional[Dict[str, Any]] = Field(
default=None,
description=(
"Per-request watermark settings: enabled, key, and context_window. "
"A key implicitly enables watermarking."
),
)
# Pre-computed prompt token IDs: when provided, bypasses chat template
# tokenization entirely. Messages are still used to derive stop tokens
@@ -1149,6 +1163,7 @@ class ChatCompletionRequest(PDRoutingFields):
"custom_params": self.custom_params,
"sampling_seed": self.seed,
"spaces_between_special_tokens": spaces_between_special_tokens,
"watermark": self.watermark,
}
if self.response_format and self.response_format.type == "json_schema":
@@ -170,6 +170,7 @@ class OpenAIServingCompletion(OpenAIServingBase):
"logit_bias": request.logit_bias,
"custom_params": request.custom_params,
"sampling_seed": request.seed,
"watermark": request.watermark,
}
# Handle response_format constraints
+31 -5
View File
@@ -8,6 +8,7 @@ from torch import nn
from sglang.kernels.ops.sampling import softmax as sampler_softmax
from sglang.kernels.ops.sampling.murmur_hash import murmur_hash32
from sglang.srt.arg_groups.choices import BUILT_IN_SAMPLING_BACKENDS
from sglang.srt.environ import envs
from sglang.srt.layers.dp_attention import (
is_dp_attention_enabled,
@@ -74,7 +75,6 @@ logger = logging.getLogger(__name__)
SYNC_TOKEN_IDS_ACROSS_TP = get_bool_env_var("SYNC_TOKEN_IDS_ACROSS_TP")
SGLANG_RETURN_ORIGINAL_LOGPROB = get_bool_env_var("SGLANG_RETURN_ORIGINAL_LOGPROB")
_CUSTOM_SAMPLER_FACTORIES: Dict[str, Callable[[], "Sampler"]] = {}
_BUILT_IN_SAMPLING_BACKENDS = {"flashinfer", "pytorch", "ascend", "intel_xpu"}
def _trace_e2e_sampler(stage: str, **fields) -> None:
@@ -147,6 +147,8 @@ class Sampler(nn.Module):
top_logprobs_nums: List[int],
token_ids_logprobs: List[List[int]],
positions: torch.Tensor,
*,
logprob_logits: Optional[torch.Tensor] = None,
):
"""Run a sampler & compute logprobs and update logits_output accordingly.
@@ -161,6 +163,7 @@ class Sampler(nn.Module):
sequence in the batch. This is used in speculative decoding.
positions: The positions of the tokens in the sequence. Used for deterministic sampling
to get the unique seed for each position.
logprob_logits: Optional pre-watermark logits for returned log probabilities.
"""
logits = logits_output.next_token_logits
maybe_detect_nan(logits, "sampler: next_token_logits")
@@ -177,6 +180,18 @@ class Sampler(nn.Module):
_trace_e2e_sampler("preprocess_enter")
logits = self._preprocess_logits(logits, sampling_info)
_trace_e2e_sampler("preprocess_returned")
target_logprobs = None
if logprob_logits is not None:
logprob_logits = self._preprocess_logits(logprob_logits, sampling_info)
if sampling_info.is_all_greedy or SGLANG_RETURN_ORIGINAL_LOGPROB:
target_logprobs = torch.nn.functional.log_softmax(
logprob_logits, dim=-1
)
else:
target_logprobs = torch.nn.functional.log_softmax(
logprob_logits / sampling_info.temperatures, dim=-1
)
target_logprobs.clamp_(min=torch.finfo(target_logprobs.dtype).min)
sampling_mask_batch_indices = sampling_info.sampling_mask_batch_indices
return_sampling_mask = sampling_mask_batch_indices is not None
sampling_support_logprobs_capture_indices = (
@@ -208,7 +223,11 @@ class Sampler(nn.Module):
)
# If requested, cache original logprobs before temperature scaling.
if return_logprob and SGLANG_RETURN_ORIGINAL_LOGPROB:
if (
return_logprob
and SGLANG_RETURN_ORIGINAL_LOGPROB
and target_logprobs is None
):
original_logprobs = torch.log_softmax(logits, dim=-1)
# In RL on-policy mode, we use log_softmax to compute logprobs to match the trainer.
@@ -259,6 +278,7 @@ class Sampler(nn.Module):
and self.enable_deterministic
and logprobs_via_logsoftmax_kernel is None
and not SGLANG_RETURN_ORIGINAL_LOGPROB
and target_logprobs is None
):
logprobs_via_logsoftmax_kernel = torch.nn.functional.log_softmax(
logits, dim=-1
@@ -278,7 +298,11 @@ class Sampler(nn.Module):
positions,
simple_sampling_case,
)
if return_logprob and not SGLANG_RETURN_ORIGINAL_LOGPROB:
if (
return_logprob
and not SGLANG_RETURN_ORIGINAL_LOGPROB
and target_logprobs is None
):
logprobs = (
logprobs_via_logsoftmax_kernel
if logprobs_via_logsoftmax_kernel is not None
@@ -287,7 +311,9 @@ class Sampler(nn.Module):
del probs
if return_logprob:
if SGLANG_RETURN_ORIGINAL_LOGPROB:
if target_logprobs is not None:
logprobs = target_logprobs
elif SGLANG_RETURN_ORIGINAL_LOGPROB:
logprobs = original_logprobs
logprob_result = self.output_logprob_processor.compute_logprobs(
logprobs,
@@ -742,7 +768,7 @@ def create_sampler(backend: Optional[str] = None) -> "Sampler":
)
return sampler
if backend is None or backend in _BUILT_IN_SAMPLING_BACKENDS:
if backend is None or backend in BUILT_IN_SAMPLING_BACKENDS:
return Sampler()
raise ValueError(
+13
View File
@@ -225,6 +225,7 @@ class GenerateReqInput:
video_config: Optional[Dict[str, Any]] = None
# The sampling_params. See descriptions below.
sampling_params: Optional[Union[List[Dict[str, Any]], Dict[str, Any]]] = None
watermark: Any = None
# Whether to return logprobs.
return_logprob: Optional[Union[List[bool], bool]] = None
# If return logprobs, the start location in the prompt for returning logprobs.
@@ -594,6 +595,7 @@ class GenerateReqInput:
self._normalize_video_data(num)
self._normalize_audio_data(num)
self._normalize_sampling_params(num)
self._normalize_watermark(num)
self._normalize_logprob_params(num)
self._normalize_return_hidden_states(num)
self._normalize_custom_logit_processor(num)
@@ -735,6 +737,16 @@ class GenerateReqInput:
else: # Already a list
self.sampling_params = self.sampling_params * self.parallel_sample_num
def _normalize_watermark(self, num):
if isinstance(self.watermark, list):
if len(self.watermark) != self.batch_size:
raise ValueError(
"The length of watermark should be equal to the batch size."
)
self.watermark = self.watermark * self.parallel_sample_num
else:
self.watermark = [self.watermark] * num
def _normalize_rid(self, num):
"""Normalize request IDs for batch processing."""
if self.rid is None:
@@ -959,6 +971,7 @@ class GenerateReqInput:
else None
),
sampling_params=self.sampling_params[i],
watermark=self.watermark[i],
return_logprob=self.return_logprob[i],
logprob_start_len=self.logprob_start_len[i],
top_logprobs_num=self.top_logprobs_num[i],
@@ -159,6 +159,11 @@ from sglang.srt.runtime_context import (
)
from sglang.srt.sampling.custom_logit_processor import supports_sampling_mask
from sglang.srt.sampling.sampling_params import SamplingParams
from sglang.srt.sampling.watermarking.core import (
redact_watermark_command_line,
redact_watermark_secrets,
resolve_watermark_request,
)
from sglang.srt.server_args import (
PortArgs,
ServerArgs,
@@ -196,6 +201,21 @@ _REQUEST_STATE_WAIT_TIMEOUT = envs.SGLANG_REQUEST_STATE_WAIT_TIMEOUT.get()
logger = logging.getLogger(__name__)
def _validate_watermark_request(sampling_params: SamplingParams) -> None:
watermark = sampling_params.watermark
features = get_exec().features
_, _, enabled = resolve_watermark_request(
watermark,
server_enabled=features.enable_watermark,
default_key=features.watermark_key,
default_context_window=features.watermark_context_window,
default_enabled=features.watermark_default_enabled,
enforce_all=features.watermark_enforce_all,
)
if enabled and (sampling_params.beam_width or 1) > 1:
raise ValueError("beam search is not supported with watermarking")
def _reject_missing_dispatched_encoder_embedding(request_obj, mm_inputs):
"""Do not silently turn a failed EPD request into local vision work."""
disagg = get_disagg()
@@ -1518,7 +1538,14 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
if self.preferred_sampling_params:
sampling_kwargs = {**self.preferred_sampling_params, **obj.sampling_params}
else:
sampling_kwargs = obj.sampling_params
sampling_kwargs = dict(obj.sampling_params)
if isinstance(obj, GenerateReqInput) and obj.watermark is not None:
if sampling_kwargs.get("watermark") is not None:
raise ValueError(
"watermark must be provided either at the request top level or "
"inside sampling_params, not both"
)
sampling_kwargs["watermark"] = obj.watermark
if isinstance(obj, GenerateReqInput) and obj.max_thinking_tokens is not None:
sampling_kwargs = dict(sampling_kwargs)
custom_params = dict(sampling_kwargs.get("custom_params") or {})
@@ -1527,6 +1554,7 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
sampling_params = self.sampling_params_class(**sampling_kwargs)
sampling_params.normalize(self.tokenizer)
sampling_params.verify(self.model_config.vocab_size)
_validate_watermark_request(sampling_params)
# Build return object
if isinstance(obj, GenerateReqInput):
@@ -2291,11 +2319,18 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
part that cannot be reconstructed afterwards.
"""
try:
return self.resolved_config_dict(self.server_args.resolved_dict())
return redact_watermark_secrets(
self.resolved_config_dict(self.server_args.resolved_dict())
)
except Exception as e:
logger.error(f"Failed to snapshot the resolved config for the dump: {e!r}")
return None
def _server_args_for_dump(self) -> Union[ServerArgs, Dict[str, Any]]:
if not get_exec().features.enable_watermark:
return self.server_args
return redact_watermark_secrets(self.server_args.resolved_dict())
def resolved_config_dict(self, base: Dict[str, Any]) -> Dict[str, Any]:
"""``base`` (a serialized ``ServerArgs``) with the control-plane changes on top."""
resolved = get_context().resolved_server_args_dict(base)
@@ -3268,10 +3303,10 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
):
logger.info(log_message)
to_dump_with_server_args = {
"server_args": self.server_args,
"config_updates": get_context().overrides_log(),
"server_args": self._server_args_for_dump(),
"config_updates": redact_watermark_secrets(get_context().overrides_log()),
"resolved_config": self._dump_config_snapshot(),
"requests": data_list.copy(),
"requests": redact_watermark_secrets(data_list.copy()),
}
def background_task():
@@ -3353,11 +3388,15 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
# Write the data to the file
data_to_dump_with_server_args = {
"server_args": self.server_args,
"config_updates": get_context().overrides_log(),
"server_args": self._server_args_for_dump(),
"config_updates": redact_watermark_secrets(
get_context().overrides_log()
),
"resolved_config": self._dump_config_snapshot(),
"requests": data_to_dump,
"launch_command": " ".join(sys.argv),
"requests": redact_watermark_secrets(data_to_dump),
"launch_command": redact_watermark_command_line(
sys.argv, self.server_args.watermark_config
),
}
with open(filename, "wb") as f:
try:
@@ -629,6 +629,8 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
lora_ids: Optional[List[str]] = None
# For dumper: request IDs for cross-step sequence tracking
rids: Optional[List[str]] = None
watermark_prompt_tail_ids: Optional[List[Optional[List[int]]]] = None
watermark_context_hash_history: Optional[List[Optional[List[int]]]] = None
# === Per-forward overrides passed explicitly to init_new ===
capture_hidden_mode: CaptureHiddenMode = None
@@ -996,6 +998,14 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
if batch.seq_lens_sum is None and seq_lens_cpu is not None:
batch.seq_lens_sum = int(seq_lens_cpu.sum())
from sglang.srt.model_executor.model_runner import ModelRunner
# Only a real ModelRunner owns watermark state; lightweight stand-ins skip it.
watermark_state = (
model_runner.watermark_state
if isinstance(model_runner, ModelRunner)
else None
)
ret = cls(
# Required core inputs
forward_mode=batch.forward_mode,
@@ -1057,6 +1067,18 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
encoder_lens_cpu=batch.encoder_lens_cpu,
lora_ids=[req.lora_id for req in batch.reqs],
rids=[req.rid for req in batch.reqs],
watermark_prompt_tail_ids=(
watermark_state.prompt_tails(batch)
if watermark_state is not None
and batch.sampling_info.has_watermark_candidates
else None
),
watermark_context_hash_history=(
watermark_state.retracted_context_hashes(batch)
if watermark_state is not None
and batch.sampling_info.has_watermark_candidates
else None
),
# Compound (carry their own device tensors)
sampling_info=batch.sampling_info,
spec_info=batch.spec_info,
@@ -201,6 +201,7 @@ from sglang.srt.runtime_context import (
)
from sglang.srt.sampling.sampling_batch_info import SamplingBatchInfo
from sglang.srt.sampling.sampling_observer import SamplingObserver
from sglang.srt.sampling.watermarking.core import WatermarkState
from sglang.srt.server_args import ( # noqa: F401 (re-export)
CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS,
ServerArgs,
@@ -310,6 +311,8 @@ def resolve_draft_attention_backend(
class ModelRunner:
"""ModelRunner runs the forward passes of the models."""
watermark_state: Optional[WatermarkState] = None
@property
def sampling_observer(self) -> Optional[SamplingObserver]:
return self._sampling_observer
@@ -615,6 +618,26 @@ class ModelRunner:
device=self.device,
)
def maybe_init_watermark_state(self):
features = get_exec().features
self.watermark_state = (
WatermarkState(
max_num_reqs=self.req_to_token_pool.req_to_token.shape[0],
context_window=features.watermark_context_window,
max_contexts_per_req=self.req_to_token_pool.req_to_token.shape[1],
vocab_size=self.model_config.vocab_size,
key=features.watermark_key,
key_b=features.watermark_key_b,
mixing_probability=features.watermark_mixing_probability,
max_probability=features.watermark_max_probability,
device=self.device,
default_enabled=features.watermark_default_enabled,
enforce_all=features.watermark_enforce_all,
)
if features.enable_watermark and not self.is_draft_worker
else None
)
def init_kv_cache_configurator(self):
# The replica count that shares this KV budget is the deployment's, not
# the one a draft scope reports; the pool is allocated outside any scope,
@@ -965,6 +988,7 @@ class ModelRunner:
# Init ngram embedding token table
self.init_ngram_embedding_manager()
self.maybe_init_watermark_state()
self.maybe_init_hisparse_coordinator()
@@ -2059,6 +2083,32 @@ class ModelRunner:
logits_output, forward_batch.sampling_info
)
watermark_state = self.watermark_state
if (
watermark_state is not None
and forward_batch.sampling_info.has_watermark_candidates
):
logprob_logits = (
logits_output.next_token_logits.clone()
if forward_batch.return_logprob
else None
)
req_pool_indices = forward_batch.req_pool_indices[
: logits_output.next_token_logits.shape[0]
]
watermark_state.init_from_prompt(
req_pool_indices,
forward_batch.watermark_prompt_tail_ids,
forward_batch.watermark_context_hash_history,
)
watermark_state.force(
logits_output.next_token_logits,
req_pool_indices,
forward_batch.sampling_info,
)
else:
logprob_logits = None
# Sample the next tokens
next_token_ids = self.sampler(
logits_output,
@@ -2072,6 +2122,9 @@ class ModelRunner:
if forward_batch.forward_mode.is_decode()
else forward_batch.seq_lens - 1
),
**(
{"logprob_logits": logprob_logits} if logprob_logits is not None else {}
),
)
if observer_state is not None:
logits_output.auxiliary_device_output = observer.after_sample(
@@ -2082,6 +2135,14 @@ class ModelRunner:
next_token_ids=next_token_ids,
forward_batch=forward_batch,
)
if (
watermark_state is not None
and forward_batch.sampling_info.has_watermark_candidates
):
watermark_state.append(
req_pool_indices,
next_token_ids,
)
return next_token_ids
def compute_logprobs_only(
@@ -16,6 +16,7 @@ from sglang.srt.runtime_context import get_exec
from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor
from sglang.srt.sampling.penaltylib.repetition_penalty import apply_scaling_penalties
from sglang.srt.sampling.sampling_params import TOP_K_ALL
from sglang.srt.sampling.watermarking.core import build_watermark_batch_config
from sglang.srt.utils.common import is_pin_memory_available
if TYPE_CHECKING:
@@ -101,6 +102,12 @@ class SamplingBatchInfo:
# Per-row top_k of mask requests (1 elsewhere); bounds the capture width.
sampling_mask_top_ks: Optional[List[int]] = None
watermark_keys: Optional[torch.Tensor] = None
watermark_context_windows: Optional[torch.Tensor] = None
watermark_enabled: Optional[torch.Tensor] = None
watermark_candidates_host: Optional[List[bool]] = None
has_watermark_candidates: bool = False
# Device
device: str = "cuda"
@@ -201,6 +208,29 @@ class SamplingBatchInfo:
sampling_mask_batch_indices = cls._make_sampling_mask_batch_indices(
return_sampling_masks, device
)
from sglang.srt.managers.schedule_batch import ScheduleBatch
features = get_exec().features
if isinstance(batch, ScheduleBatch) and features.enable_watermark:
watermark_config = build_watermark_batch_config(
reqs,
default_key=features.watermark_key,
default_context_window=features.watermark_context_window,
default_enabled=features.watermark_default_enabled,
enforce_all=features.watermark_enforce_all,
device=device,
)
watermark_keys = watermark_config.keys
watermark_context_windows = watermark_config.context_windows
watermark_enabled = watermark_config.enabled
watermark_candidates_host = watermark_config.candidates_host
has_watermark_candidates = watermark_config.has_candidates
else:
watermark_keys = None
watermark_context_windows = None
watermark_enabled = None
watermark_candidates_host = None
has_watermark_candidates = False
sampling_support_logprobs_capture_indices = (
cls._make_sampling_support_logprobs_capture_indices(
return_sampling_masks,
@@ -283,6 +313,11 @@ class SamplingBatchInfo:
return_sampling_masks=return_sampling_masks,
return_sampling_support_logprobs=return_sampling_support_logprobs,
sampling_mask_batch_indices=sampling_mask_batch_indices,
watermark_keys=watermark_keys,
watermark_context_windows=watermark_context_windows,
watermark_enabled=watermark_enabled,
watermark_candidates_host=watermark_candidates_host,
has_watermark_candidates=has_watermark_candidates,
sampling_support_logprobs_capture_indices=(
sampling_support_logprobs_capture_indices
),
@@ -440,11 +475,20 @@ class SamplingBatchInfo:
"top_ks",
"min_ps",
"sampling_seed",
"watermark_keys",
"watermark_context_windows",
"watermark_enabled",
]:
value = getattr(self, item, None)
if value is not None:
setattr(self, item, value[keep_indices_device])
if self.watermark_candidates_host is not None:
self.watermark_candidates_host = [
self.watermark_candidates_host[index] for index in keep_indices
]
self.has_watermark_candidates = any(self.watermark_candidates_host)
if self.logit_bias is not None:
self.logit_bias = self.logit_bias[keep_indices_device]
if self.return_sampling_masks is not None:
@@ -570,12 +614,22 @@ class SamplingBatchInfo:
"top_ks",
"min_ps",
"sampling_seed",
"watermark_keys",
"watermark_context_windows",
"watermark_enabled",
]:
self_val = getattr(self, item, None)
other_val = getattr(other, item, None)
if self_val is not None and other_val is not None:
setattr(self, item, torch.cat([self_val, other_val]))
if (
self.watermark_candidates_host is not None
and other.watermark_candidates_host is not None
):
self.watermark_candidates_host.extend(other.watermark_candidates_host)
self.has_watermark_candidates = any(self.watermark_candidates_host)
self.is_all_greedy &= other.is_all_greedy
self.is_any_greedy |= other.is_any_greedy
self.need_top_p_sampling |= other.need_top_p_sampling
@@ -19,6 +19,11 @@ from typing import Dict, List, Optional, Sequence, Set, Union
import msgspec
from sglang.srt.sampling.watermarking.core import (
WatermarkRequestConfig,
normalize_watermark_request,
)
# sre_parse is deprecated in Python 3.11+, use re._parser instead
try:
import re._parser as sre_parse
@@ -150,6 +155,7 @@ class SamplingParams(msgspec.Struct, kw_only=True, array_like=True):
logit_bias: Optional[Dict[str, float]] = None
sampling_seed: Optional[int] = None
custom_params: Optional[Dict[str, CustomParamValue]] = None
watermark: Optional[WatermarkRequestConfig] = None
# --- Internal fields (populated by __post_init__ or normalize(), not API-facing) ---
stop_strs: Optional[Union[str, List[str]]] = None # from stop
@@ -204,6 +210,7 @@ class SamplingParams(msgspec.Struct, kw_only=True, array_like=True):
self.no_stop_trim = (
self.no_stop_trim if self.no_stop_trim is not None else False
)
self.watermark = normalize_watermark_request(self.watermark)
# An empty grammar constraint means "unset", not "constrain to nothing".
self.json_schema = self.json_schema or None
@@ -0,0 +1,171 @@
# Text watermarking design
This package implements keyed Aaronson-Gumbel sampling and model-free detection for generated text. Watermarking is opt-in, preserves the target distribution under the ideal hash model, and supports ordinary and speculative decoding. See the [operator guide](../../../../../docs/docs/advanced_features/text_watermarking.mdx) for deployment guidance and the full compatibility matrix.
## Algorithm
After penalties, logit bias, grammar masking, temperature scaling, and top-k/top-p/min-p truncation, let `p[t]` be the probability of token `t`. For each supported token, the selector derives a deterministic uniform `u[t]` from the request key and preceding context, then selects
```text
argmax_t log(u[t]) / p[t]
```
This is the logarithmic form of `argmax_t u[t]^(1 / p[t])`. Independent uniform values select each token with probability `p[t]`; masked and truncated tokens have zero support and cannot win. Ties select the lowest token ID.
All hash arithmetic wraps to unsigned 32 bits. `mix` and `fmix32` are the MurmurHash3 x86 32-bit body mix and finalizer:
```text
mix(state, value):
value = value * 0xcc9e2d51
value = rotl32(value, 15)
value = value * 0x1b873593
state = rotl32(state XOR value, 13)
return state * 5 + 0xe6546b64
fmix32(value):
value = value XOR (value >> 16)
value = value * 0x85ebca6b
value = value XOR (value >> 13)
value = value * 0xc2b2ae35
return value XOR (value >> 16)
```
A key is 1 to 16 hexadecimal digits with an optional `0x` prefix, interpreted as an unsigned 64-bit integer. For context tokens `c_1 ... c_L`, candidate token `t`, and the low and high 32-bit key words:
```text
context_hash = fmix32(mix(...mix(mix(0, c_1), c_2)..., c_L) XOR (4 * L))
token_hash = fmix32(mix(mix(mix(mix(0, key_lo), key_hi), context_hash), t) XOR 16)
u = (token_hash + 0.5) / 2^32
```
The context is the last `h` prompt and committed output tokens before the sampled position. The first generated token therefore uses the prompt tail.
Direct fp32 evaluation of `u` rounds the largest 128 hash values to `1.0`. The Triton and PyTorch selectors preserve relative precision near one by using the direct expression below `2^31` and
```text
log1p(-(2^32 - token_hash - 0.5) / 2^32)
```
above it. The integer complement is formed before conversion to floating point.
## Detection
Detection needs token IDs, the same key and context window, and preferably the exact prompt token IDs. For each generated token with uniform value `u`, it adds `-log(1 - u)` to the score. Under the null hypothesis, the sum over `n` contexts follows `Gamma(shape=n, scale=1)`. The detector reports the one-sided upper-tail p-value, its logarithm, and `z = (score - n) / sqrt(n)`.
The detector scores each exact context tuple once. Runtime uses the 32-bit context hash for admission; a rare collision can only add an ordinary null term to detector scoring and dilute the signal. Without prompt IDs, positions whose initial context is unknown are skipped.
Scoring uses a fixed prefix of the first `min(4,096, per-request token-pool capacity)` distinct contexts. Later positions are ordinary samples after runtime reaches the same history budget, so including them would dilute the test. The model-free detector cannot identify entropy-gated or greedy positions; it scores them as conservative null terms.
The [operator guide's detector specification](../../../../../docs/docs/advanced_features/text_watermarking.mdx#detection) is normative for prompt alignment, prefix handling, dual-key partitioning, and result interpretation.
## Runtime design
- **Policy resolution.** Server policy and each request's `watermark` object resolve to a key, context window, and enabled bit before batch construction. Mixed batches retain independent configuration and state per row.
- **Ring buffer.** Each `req_pool_idx` owns the committed prompt/output tail used to reconstruct contexts. GPU state avoids stale CPU output IDs under overlap scheduling and supplies current contexts for every speculative tree row.
- **Repeated-context masking.** A context hash is forced at most once per request. Repeated contexts use ordinary sampling, preventing a deterministic short cycle from repeatedly excluding EOS. Only forced contexts enter the history.
- **Greedy bypass.** Rows normalized to `top_k <= 1`, including `temperature=0`, use ordinary greedy selection and are not recorded.
- **Entropy gate.** A row uses ordinary sampling when its largest truncated probability exceeds the server's `max_probability`. Skipped contexts remain eligible for a later higher-entropy occurrence.
- **Speculative decoding.** NGRAM, EAGLE, EAGLE3, and NEXTN reconstruct verify-row contexts from the committed ring tail, tree mask, draft tokens, and positions. Normal decode preserves a clone for logprobs while speculative decode forces a clone, so both report the pre-watermark target distribution. Fused kernels admit contexts and append only accepted tokens.
- **Selector paths.** Finite `top_k <= 8,192` uses `torch.topk`, canonicalizes boundary ties, and performs small-k truncation. Unlimited top-k retains full-vocabulary sorting. Both paths use the same hash and selection contract.
- **Disabled-batch exit.** A host-side candidate bit follows filtering and merging. Fully disabled or greedy-only batches skip prompt initialization, verify cloning, selector work, and append kernels.
- **Retraction and slot reuse.** Re-prefill rebuilds the distinct-context history from accepted output. A new request resets all state for a reused request-pool slot.
- **Dual key.** A deterministic per-context coin chooses key A with probability `P` and key B otherwise:
```text
coin = fmix32(mix(mix(mix(mix(mix(0, a_lo), a_hi), b_lo), b_hi), context_hash) XOR 20)
threshold = floor(P * 2^32)
key = key A if coin < threshold else key B
```
A request key replaces key A; key B remains server-owned. Omitting key B leaves the single-key path unchanged.
## Package layout
| File | Role |
| --- | --- |
| `core.py` | Request policy resolution, batch configuration, ring/history state, PyTorch reference selection, and speculative orchestration. |
| `config.py` | Key parsing, bounded JSON configuration loading, and shared protocol limits. |
| `detector.py` | Torch-free single-key and dual-key reference detector. |
| `__init__.py` | Minimal public detector exports. |
The Triton selector and state kernels live in `sglang/kernels/ops/sampling/textseal_selector.py`.
## Server configuration
`--enable-watermark` allocates CUDA watermark state and accepts request configuration. All other settings come from one JSON object passed with `--watermark-config`, as a file path or inline JSON (a value starting with `{`); every field is optional and unknown fields are rejected.
| Field | Default | Behavior |
| --- | --- | --- |
| `key` | unset | Server key A. |
| `key_b` | unset | Enables dual-key mode with server key B. Requires key A. |
| `context_window` | `4` | Default context window and per-request maximum, from 1 to 64. |
| `mixing_probability` | `0.5` | Probability of selecting key A for an eligible context; a non-default value requires `key_b`. |
| `max_probability` | `1.0` | Uses ordinary sampling when the largest truncated probability exceeds this value. |
| `default_enabled` | `false` | Uses the server key when a request omits `watermark`; explicit opt-out remains allowed. |
| `enforce_all` | `false` | Uses the server key by default and rejects explicit opt-out. |
Default-on and enforce-all modes require a server key. A server key without either mode remains available only to requests that opt in.
Use a regular config file readable only by the server account for keys; inline JSON is visible in the process command line. Launch-command, request-log, server-info, and crash-payload rendering redacts key fields and the `--watermark-config` value. Process command lines, client request bodies, and process or GPU core dumps remain separate secret-bearing surfaces.
## Per-request control
The native `/generate` endpoint and OpenAI-compatible completion endpoints accept a `watermark` object:
| Field | Type | Behavior |
| --- | --- | --- |
| `enabled` | boolean | `true` uses the server key; `false` opts out unless enforce-all is active. |
| `key` | hex string | Replaces key A and implicitly enables watermarking. |
| `context_window` | integer | Selects a window from 1 through the server maximum. |
| Server mode | Omitted | `{"enabled": false}` | `{"enabled": true}` | `{"key": "HEX"}` |
| --- | --- | --- | --- | --- |
| Watermarking disabled | Off | Off | HTTP 400 | HTTP 400 |
| Enabled, no server key | Off | Off | HTTP 400 | Request key |
| Enabled with server key | Off | Off | Server key | Request key |
| Default-enabled | Server key | Off | Server key | Request key |
| Enforce-all | Server key | HTTP 400 | Server key | Request key |
A request is rejected when it asks for an unavailable key, combines a key with `enabled=false`, exceeds the server context-window limit, or uses a fail-closed incompatible mode. On `/generate`, place `watermark` at the request top level or in `sampling_params`, not both.
## Using the detector
Within an SGLang environment:
```python
from sglang.srt.sampling.watermarking import WatermarkDetector
detector = WatermarkDetector("0123456789abcdef", context_window=4)
result = detector.detect_tokens(completion_ids, prompt_token_ids=prompt_ids)
print(result.combined.num_contexts, result.combined.z_score, result.combined.p_value)
```
For dual-key output, pass `key_b` and the server's `mixing_probability`. Exact generated token IDs are preferred because decoding and re-tokenizing text need not reproduce the original sequence.
The detector package imports only the Python standard library and `msgspec`; NumPy token arrays are accepted but NumPy is optional. It can be copied into another application without SGLang or PyTorch:
```bash
cp -R python/sglang/srt/sampling/watermarking detector_app/watermarking
python -m pip install msgspec
```
```python
from watermarking.detector import WatermarkDetector
detector = WatermarkDetector("0123456789abcdef", context_window=4)
result = detector.detect_tokens(completion_ids, prompt_token_ids=prompt_ids)
```
## Limitations
- Greedy generation carries no watermark signal. Short and low-entropy text provides less statistical power; the entropy gate trades signal for ordinary sampling at low-entropy positions.
- Paraphrasing and translation erase the signal. Edits reduce the number of usable contexts.
- The 32-bit Murmur-derived hash is not a cryptographic PRF. A key holder can detect and forge the watermark.
- Detection requires the generation key, context window, tokenizer, and preferably the exact prompt and completion token IDs.
- Generation stops forcing after its context-history budget. Rare 32-bit context-hash collisions and conservative cross-sibling suppression in wide speculative trees can only reduce signal.
- Unlimited top-k sorts the full vocabulary. Its throughput cost and speculative overhead depend on model, hardware, and workload.
- Unsupported combinations fail closed. The operator guide maintains the compatibility matrix.
## Attribution
The fused Triton selector is adapted from Meta's [TextSeal](https://github.com/facebookresearch/textseal) selector implementation under Apache-2.0 and reimplemented on the hash contract above. Detection follows this contract rather than TextSeal's detector tooling.
@@ -0,0 +1,13 @@
from .detector import (
WatermarkDetection,
WatermarkDetector,
WatermarkStatistics,
detect,
)
__all__ = [
"WatermarkDetection",
"WatermarkDetector",
"WatermarkStatistics",
"detect",
]
@@ -0,0 +1,117 @@
from __future__ import annotations
import json
import logging
import os
import stat
from pathlib import Path
from typing import Any
import msgspec
logger = logging.getLogger(__name__)
_MAX_CONFIG_BYTES = 4096
MAX_WATERMARK_CONTEXT_WINDOW = 64
MAX_WATERMARKED_CONTEXTS_PER_REQUEST = 4096
class WatermarkConfigError(ValueError):
pass
class WatermarkServerConfig(msgspec.Struct, frozen=True, kw_only=True):
# None marks a field the file leaves unset, so server defaults still apply.
key: str | None = None
key_b: str | None = None
context_window: int | None = None
mixing_probability: float | None = None
max_probability: float | None = None
default_enabled: bool | None = None
enforce_all: bool | None = None
def __repr__(self) -> str:
return (
"WatermarkServerConfig(key=<redacted>, key_b=<redacted>, "
f"context_window={self.context_window!r}, "
f"mixing_probability={self.mixing_probability!r}, "
f"max_probability={self.max_probability!r}, "
f"default_enabled={self.default_enabled!r}, "
f"enforce_all={self.enforce_all!r})"
)
def parse_watermark_key(value: Any) -> int:
if not isinstance(value, str):
raise ValueError("watermark key must be a hex string")
digits = value[2:] if value.lower().startswith("0x") else value
if not 1 <= len(digits) <= 16:
raise ValueError("watermark key must contain 1 to 16 hex digits")
if any(character not in "0123456789abcdefABCDEF" for character in digits):
raise ValueError("watermark key must contain only hex digits")
key = int(digits, 16)
return key if key < (1 << 63) else key - (1 << 64)
def _read_config_file(path: str) -> str:
config_path = Path(path).expanduser()
try:
flags = os.O_RDONLY | os.O_CLOEXEC | os.O_NONBLOCK
descriptor = os.open(config_path, flags)
with os.fdopen(descriptor, encoding="utf-8") as config_file:
file_stat = os.fstat(config_file.fileno())
if not stat.S_ISREG(file_stat.st_mode):
raise WatermarkConfigError(
"watermark config must be a readable regular file"
)
if stat.S_IMODE(file_stat.st_mode) & 0o077:
logger.warning(
"watermark config is readable by group or other users; "
"restrict it to the server account"
)
return config_file.read(_MAX_CONFIG_BYTES + 1)
except (OSError, UnicodeError):
raise WatermarkConfigError("failed to read watermark config JSON") from None
def load_watermark_config(source: str) -> WatermarkServerConfig:
if source.lstrip().startswith("{"):
payload = source
else:
payload = _read_config_file(source)
if len(payload.encode("utf-8")) > _MAX_CONFIG_BYTES:
raise WatermarkConfigError("watermark config exceeds 4096 bytes")
try:
raw = json.loads(payload)
except json.JSONDecodeError:
raise WatermarkConfigError("failed to read watermark config JSON") from None
if not isinstance(raw, dict):
raise WatermarkConfigError("watermark config must be a JSON object")
if set(raw) - set(WatermarkServerConfig.__struct_fields__):
raise WatermarkConfigError("watermark config contains unknown fields")
try:
for name in ("key", "key_b"):
if raw.get(name) is not None:
parse_watermark_key(raw[name])
except ValueError as error:
raise WatermarkConfigError(str(error)) from error
context_window = raw.get("context_window")
if context_window is not None and (
isinstance(context_window, bool)
or not isinstance(context_window, int)
or not 1 <= context_window <= MAX_WATERMARK_CONTEXT_WINDOW
):
raise WatermarkConfigError(
"watermark config context_window must be an integer from 1 to 64"
)
for name in ("mixing_probability", "max_probability"):
value = raw.get(name)
if value is not None and (
isinstance(value, bool) or not isinstance(value, (int, float))
):
raise WatermarkConfigError(f"watermark config {name} must be a number")
for name in ("default_enabled", "enforce_all"):
if raw.get(name) is not None and not isinstance(raw[name], bool):
raise WatermarkConfigError(f"watermark config {name} must be a boolean")
return WatermarkServerConfig(**raw)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,247 @@
from __future__ import annotations
import math
from numbers import Integral, Real
from typing import Iterable, Optional, Protocol, Sequence
import msgspec
from .config import (
MAX_WATERMARK_CONTEXT_WINDOW,
MAX_WATERMARKED_CONTEXTS_PER_REQUEST,
parse_watermark_key,
)
_MASK32 = 0xFFFFFFFF
_UINT32_SCALE = 1 << 32
def _rotl32(value: int, shift: int) -> int:
return ((value << shift) | (value >> (32 - shift))) & _MASK32
def _murmur3_words(words: Sequence[int]) -> int:
state = 0
for word in words:
value = (int(word) * 0xCC9E2D51) & _MASK32
value = (_rotl32(value, 15) * 0x1B873593) & _MASK32
state = (_rotl32(state ^ value, 13) * 5 + 0xE6546B64) & _MASK32
state ^= 4 * len(words)
state ^= state >> 16
state = (state * 0x85EBCA6B) & _MASK32
state ^= state >> 13
state = (state * 0xC2B2AE35) & _MASK32
return state ^ (state >> 16)
def hash_context(token_ids: Sequence[int]) -> int:
return _murmur3_words([int(token_id) & _MASK32 for token_id in token_ids])
def watermark_hash(key: int, context_hash: int, token_id: int) -> int:
return _murmur3_words(
(key & _MASK32, (key >> 32) & _MASK32, context_hash, token_id)
)
def position_coin(key_a: int, key_b: int, context_hash: int) -> int:
return _murmur3_words(
(
key_a & _MASK32,
(key_a >> 32) & _MASK32,
key_b & _MASK32,
(key_b >> 32) & _MASK32,
context_hash,
)
)
def _gamma_log_survival(score: float, shape: int) -> float:
if score == 0:
return 0.0
log_score = math.log(score)
terms = [k * log_score - math.lgamma(k + 1) for k in range(shape)]
peak = max(terms)
tail = -score + peak + math.log(math.fsum(math.exp(t - peak) for t in terms))
return min(0.0, tail)
class WatermarkStatistics(msgspec.Struct, frozen=True):
num_contexts: int
score: float
z_score: float
p_value: float
# Survives when p_value underflows to 0.0 on long watermarked texts.
log_p_value: float
@classmethod
def from_scores(cls, scores: Sequence[float]) -> WatermarkStatistics:
n = len(scores)
if n == 0:
return cls(
num_contexts=0, score=0.0, z_score=0.0, p_value=1.0, log_p_value=0.0
)
score = math.fsum(scores)
log_p = _gamma_log_survival(score, n)
return cls(
num_contexts=n,
score=score,
z_score=(score - n) / math.sqrt(n),
p_value=math.exp(log_p),
log_p_value=log_p,
)
class WatermarkDetection(msgspec.Struct, frozen=True, kw_only=True):
combined: WatermarkStatistics
key_a_all_positions: WatermarkStatistics
key_b_all_positions: Optional[WatermarkStatistics]
key_a_partition: Optional[WatermarkStatistics]
key_b_partition: Optional[WatermarkStatistics]
prefix_limit_reached: bool
tokens_examined: int
repeated_contexts: int
skipped_initial_tokens: int
class Tokenizer(Protocol):
def encode(self, text: str, *, add_special_tokens: bool) -> list[int]: ...
def _validate_token_ids(values: Iterable[int]) -> list[int]:
token_ids = []
for value in values:
if (
isinstance(value, bool)
or not isinstance(value, Integral)
or not 0 <= value <= _MASK32
):
raise ValueError("token IDs must be unsigned 32-bit integers")
token_ids.append(int(value))
return token_ids
def _validate_int_range(name: str, value: int, limit: int) -> int:
if (
isinstance(value, bool)
or not isinstance(value, Integral)
or not 1 <= value <= limit
):
raise ValueError(f"{name} must be an integer from 1 to {limit}")
return int(value)
class WatermarkDetector:
def __init__(
self,
key: str,
*,
key_b: Optional[str] = None,
mixing_probability: float = 0.5,
context_window: int = 4,
max_contexts: int = MAX_WATERMARKED_CONTEXTS_PER_REQUEST,
) -> None:
if (
isinstance(mixing_probability, bool)
or not isinstance(mixing_probability, Real)
or not 0 < mixing_probability < 1
):
raise ValueError(
"watermark mixing probability must be strictly between 0 and 1"
)
self._key_a = parse_watermark_key(key)
self._key_b = parse_watermark_key(key_b) if key_b is not None else None
self._mixing_threshold = int(mixing_probability * _UINT32_SCALE)
self.context_window = _validate_int_range(
"context_window", context_window, MAX_WATERMARK_CONTEXT_WINDOW
)
self.max_contexts = _validate_int_range(
"max_contexts", max_contexts, MAX_WATERMARKED_CONTEXTS_PER_REQUEST
)
def detect_tokens(
self,
token_ids: Iterable[int],
*,
prompt_token_ids: Iterable[int] = (),
) -> WatermarkDetection:
prompt = _validate_token_ids(prompt_token_ids)
completion = _validate_token_ids(token_ids)
tokens = prompt[-self.context_window :] + completion
start = min(len(prompt), self.context_window)
# Without prompt IDs, the first h contexts would include unknown prompt tokens.
first = start if prompt else min(self.context_window, len(completion))
seen = set()
scores_a, scores_b, partition_a, partition_b = [], [], [], []
repeated = 0
examined = first - start
for position in range(first, len(tokens)):
examined = position - start + 1
context = tuple(tokens[max(0, position - self.context_window) : position])
if context in seen:
repeated += 1
continue
seen.add(context)
context_hash = hash_context(context)
score_a = self._score(self._key_a, context_hash, tokens[position])
scores_a.append(score_a)
if self._key_b is not None:
score_b = self._score(self._key_b, context_hash, tokens[position])
scores_b.append(score_b)
coin = position_coin(self._key_a, self._key_b, context_hash)
if coin < self._mixing_threshold:
partition_a.append(score_a)
else:
partition_b.append(score_b)
if len(seen) == self.max_contexts:
break
dual = self._key_b is not None
stats = WatermarkStatistics.from_scores
return WatermarkDetection(
combined=stats(partition_a + partition_b if dual else scores_a),
key_a_all_positions=stats(scores_a),
key_b_all_positions=stats(scores_b) if dual else None,
key_a_partition=stats(partition_a) if dual else None,
key_b_partition=stats(partition_b) if dual else None,
prefix_limit_reached=len(seen) == self.max_contexts,
tokens_examined=examined,
repeated_contexts=repeated,
skipped_initial_tokens=first - start,
)
def detect_text(
self,
text: str,
tokenizer: Tokenizer,
*,
prompt_token_ids: Iterable[int] = (),
) -> WatermarkDetection:
return self.detect_tokens(
tokenizer.encode(text, add_special_tokens=False),
prompt_token_ids=prompt_token_ids,
)
@staticmethod
def _score(key: int, context_hash: int, token_id: int) -> float:
uniform = (watermark_hash(key, context_hash, token_id) + 0.5) / _UINT32_SCALE
return -math.log1p(-uniform)
def detect(
token_ids: Iterable[int],
*,
key: str,
key_b: Optional[str] = None,
prompt_token_ids: Iterable[int] = (),
mixing_probability: float = 0.5,
context_window: int = 4,
) -> WatermarkDetection:
detector = WatermarkDetector(
key,
key_b=key_b,
mixing_probability=mixing_probability,
context_window=context_window,
)
return detector.detect_tokens(token_ids, prompt_token_ids=prompt_token_ids)
+16 -2
View File
@@ -300,9 +300,18 @@ class ServerArgs:
@property
def launch_command(self) -> str | None:
"""Original CLI arguments or Engine constructor call; ``None`` for direct construction."""
"""Original CLI arguments or Engine call, with secrets redacted."""
return getattr(self, "_launch_command", None)
def __repr__(self) -> str:
hidden = {"watermark_key", "watermark_key_b", "watermark_config"}
values = ", ".join(
f"{field.name}={getattr(self, field.name)!r}"
for field in record_fields(type(self))
if field.name not in hidden
)
return f"ServerArgs({values})"
def resolved_dict(self) -> dict[str, Any]:
"""Serialize resolved field values, expanding nested records and excluding bookkeeping.
@@ -688,7 +697,12 @@ def prepare_server_args(argv: list[str]) -> ServerArgs:
# Not a field: the record's fields are the configuration, and this is how
# the configuration was asked for. It rides along on the record so a
# subprocess copy can answer the same question the launcher can.
server_args._launch_command = " ".join(argv)
# lazy: watermark imports torch, outside server_args' top-level boundary.
from sglang.srt.sampling.watermarking.core import redact_watermark_command_line
server_args._launch_command = redact_watermark_command_line(
argv, raw_args.watermark_config
)
return server_args
+54 -1
View File
@@ -40,6 +40,7 @@ if TYPE_CHECKING:
from sglang.srt.mem_cache.memory_pool import ReqToTokenPool
from sglang.srt.model_executor.model_runner import ModelRunner
from sglang.srt.sampling.sampling_batch_info import SamplingBatchInfo
from sglang.srt.sampling.watermarking.core import WatermarkState
from sglang.srt.speculative.eagle_info import EagleVerifyInput
_is_cuda = is_cuda()
@@ -741,6 +742,9 @@ def eagle_sample(
logits_output: LogitsProcessorOutput,
grammar_mask: Optional[GrammarMask] = None,
uno_target_max_top_k: Optional[int] = None,
*,
watermark_state: Optional[WatermarkState] = None,
watermark_full_mask: bool = True,
):
"""
Verify and find accepted tokens based on logits output and batch
@@ -801,6 +805,46 @@ def eagle_sample(
if grammar_mask is not None:
grammar_mask.apply(next_token_logits)
watermark_context_hashes = None
watermark_selected = None
if watermark_state is not None and sampling_info.has_watermark_candidates:
if batch.return_logprob:
# compute_spec_logprobs reads logits_output after verify.
next_token_logits = next_token_logits.clone()
if next_token_logits.is_cuda:
watermark_context_hashes, watermark_selected = (
watermark_state.force_speculative_from_tree(
logits=next_token_logits,
req_pool_indices=batch.req_pool_indices,
draft_tokens=verify_input.draft_token,
custom_mask=verify_input.custom_mask,
positions=verify_input.positions,
sampling_info=sampling_info,
draft_token_num=verify_input.draft_token_num,
full_mask=watermark_full_mask,
)
)
else:
contexts, context_lengths = watermark_state.speculative_contexts(
req_pool_indices=batch.req_pool_indices,
draft_tokens=verify_input.draft_token,
custom_mask=verify_input.custom_mask,
positions=verify_input.positions,
draft_token_num=verify_input.draft_token_num,
full_mask=watermark_full_mask,
context_windows=watermark_state.context_windows(sampling_info),
)
watermark_context_hashes, watermark_selected = (
watermark_state.force_speculative(
logits=next_token_logits,
req_pool_indices=batch.req_pool_indices,
contexts=contexts,
context_lengths=context_lengths,
sampling_info=sampling_info,
draft_token_num=verify_input.draft_token_num,
)
)
candidates = verify_input.draft_token.reshape(bs, verify_input.draft_token_num)
predict_shape = list(next_token_logits.shape)[:-1]
predict = torch.zeros(predict_shape, dtype=torch.int32, device=device).flatten()
@@ -1051,7 +1095,16 @@ def eagle_sample(
# `num_correct_drafts` stays drafts-only inside this function; the returned
# tensor includes the trailing/bonus token via out-of-place +1 so the
# name no longer flips semantics mid-function (naming doc C2).
return predict, num_correct_drafts + 1, accept_index
accept_lens = num_correct_drafts + 1
if watermark_state is not None and sampling_info.has_watermark_candidates:
watermark_state.record_speculative(
batch.req_pool_indices,
watermark_context_hashes,
watermark_selected,
accept_index,
accept_lens,
)
return predict, accept_lens, accept_index
def eagle_prepare_for_decode(batch: ScheduleBatch):
@@ -701,6 +701,12 @@ def run_eagle_verify(
logits_output,
grammar_mask,
uno_target_max_top_k=uno_target_max_top_k,
watermark_state=target_worker.model_runner.watermark_state,
watermark_full_mask=(
target_worker.model_runner.attn_backend.verify_mask is None
or target_worker.model_runner.attn_backend.verify_mask.mode
== TreeMaskMode.FULL_MASK
),
)
fused_commit_outputs = predict.is_cuda and not batch.forward_mode.is_idle()
prepared_commit_steps = prepared_draft_inputs = None
@@ -751,6 +757,15 @@ def run_eagle_verify(
if not fused_commit_outputs and not batch.forward_mode.is_idle():
accept_tokens = predict[accept_index]
if (
target_worker.model_runner.watermark_state is not None
and batch.sampling_info.has_watermark_candidates
):
target_worker.model_runner.watermark_state.append_speculative(
batch.req_pool_indices,
accept_tokens,
accept_lens,
)
bonus_tokens = torch.empty_like(accept_lens, dtype=torch.int32)
# stride = accept_tokens per-req width = accept_index.shape[1]
# (spec_steps + 1); NOT num_draft_tokens, wrong for topk > 1 trees.
+18 -2
View File
@@ -479,7 +479,13 @@ class NGRAMWorker(BaseSpecWorker):
predict,
accept_lens,
accept_index,
) = eagle_sample(verify_input, batch, logits_output, grammar_mask)
) = eagle_sample(
verify_input,
batch,
logits_output,
grammar_mask,
watermark_state=self.model_runner.watermark_state,
)
new_seq_lens = batch.seq_lens + accept_lens
commit_mamba_states_after_verify(
self.target_worker,
@@ -488,7 +494,17 @@ class NGRAMWorker(BaseSpecWorker):
accept_index,
self.draft_token_num,
)
accept_tokens = predict[accept_index].flatten()
accept_tokens = predict[accept_index]
if (
self.model_runner.watermark_state is not None
and batch.sampling_info.has_watermark_candidates
):
self.model_runner.watermark_state.append_speculative(
batch.req_pool_indices,
accept_tokens,
accept_lens,
)
accept_tokens = accept_tokens.flatten()
next_token_ids = accept_tokens
# The KV mover expects drafts-only counts. NGRAM's
+4 -1
View File
@@ -3660,13 +3660,16 @@ def start_follower_grpc_server(server_args, scheduler_info: Dict[str, Any]):
return None
from sglang.srt.rust_extensions import load_rust_extension
from sglang.srt.sampling.watermarking.core import redact_watermark_secrets
grpc_native = load_rust_extension("sglang.srt.rust_extensions._grpc")
return grpc_native.start_metadata_server(
host=serving.host,
port=serving.grpc_port,
server_info_json=json.dumps(
msgspec_to_builtins(build_server_info(server_args, scheduler_info)),
msgspec_to_builtins(
redact_watermark_secrets(build_server_info(server_args, scheduler_info))
),
default=str,
),
)
+17 -4
View File
@@ -18,6 +18,7 @@ import logging
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set, Tuple, Union
from sglang.srt.environ import envs
from sglang.srt.sampling.watermarking.core import redact_watermark_secrets
from sglang.srt.utils.log_utils import create_log_targets, log_json
if TYPE_CHECKING:
@@ -252,10 +253,15 @@ def _dataclass_to_string_truncated(
elif isinstance(data, (list, tuple)):
if len(data) > max_length:
half_length = max_length // 2
return str(data[:half_length]) + " ... " + str(data[-half_length:])
return (
str(redact_watermark_secrets(data[:half_length]))
+ " ... "
+ str(redact_watermark_secrets(data[-half_length:]))
)
else:
return str(data)
return str(redact_watermark_secrets(data))
elif isinstance(data, dict):
data = redact_watermark_secrets(data)
return (
"{"
+ ", ".join(
@@ -267,10 +273,13 @@ def _dataclass_to_string_truncated(
)
elif dataclasses.is_dataclass(data):
fields = dataclasses.fields(data)
values = redact_watermark_secrets(
{f.name: getattr(data, f.name) for f in fields}
)
return (
f"{data.__class__.__name__}("
+ ", ".join(
f"{f.name}={_dataclass_to_string_truncated(getattr(data, f.name), max_length)}"
f"{f.name}={_dataclass_to_string_truncated(values[f.name], max_length)}"
for f in fields
if f.name not in skip_names
)
@@ -296,6 +305,7 @@ def _transform_data_for_logging(
return list(data[:half_length]) + ["..."] + list(data[-half_length:])
return [_transform_data_for_logging(v, max_length) for v in data]
elif isinstance(data, dict):
data = redact_watermark_secrets(data)
return {
k: _transform_data_for_logging(v, max_length)
for k, v in data.items()
@@ -303,8 +313,11 @@ def _transform_data_for_logging(
}
elif dataclasses.is_dataclass(data):
fields = dataclasses.fields(data)
values = redact_watermark_secrets(
{f.name: getattr(data, f.name) for f in fields}
)
return {
f.name: _transform_data_for_logging(getattr(data, f.name), max_length)
f.name: _transform_data_for_logging(values[f.name], max_length)
for f in fields
if f.name not in skip_names
}
@@ -304,6 +304,17 @@ pub struct SamplingParams {
/// the number of returned sequences (n \<= beam_width).
#[prost(int64, optional, tag = "26")]
pub beam_width: ::core::option::Option<i64>,
#[prost(message, optional, tag = "27")]
pub watermark: ::core::option::Option<WatermarkRequestConfig>,
}
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
pub struct WatermarkRequestConfig {
#[prost(bool, optional, tag = "1")]
pub enabled: ::core::option::Option<bool>,
#[prost(string, optional, tag = "2")]
pub key: ::core::option::Option<::prost::alloc::string::String>,
#[prost(int64, optional, tag = "3")]
pub context_window: ::core::option::Option<i64>,
}
/// (message_json).unknown_fields defaults to IGNORE — stated here as schema
/// intent, load-bearing for client compat.
@@ -1107,6 +1107,9 @@ impl ::serde::Serialize for SamplingParams {
if let Some(v) = &self.beam_width {
::serde::ser::SerializeMap::serialize_entry(&mut map, "beam_width", v)?;
}
if let Some(v) = &self.watermark {
::serde::ser::SerializeMap::serialize_entry(&mut map, "watermark", v)?;
}
::serde::ser::SerializeMap::end(map)
}
}
@@ -1140,6 +1143,7 @@ impl<'de> ::serde::Deserialize<'de> for SamplingParams {
"sampling_seed",
"custom_params",
"beam_width",
"watermark",
];
struct V;
impl<'de> ::serde::de::Visitor<'de> for V {
@@ -1177,6 +1181,7 @@ impl<'de> ::serde::Deserialize<'de> for SamplingParams {
let mut f_sampling_seed = None;
let mut f_custom_params = None;
let mut f_beam_width = None;
let mut f_watermark = None;
while let Some(key) = map.next_key::<::std::borrow::Cow<'_, str>>()? {
match key.as_ref() {
"max_new_tokens" => {
@@ -1383,6 +1388,12 @@ impl<'de> ::serde::Deserialize<'de> for SamplingParams {
}
f_beam_width = Some(map.next_value::<Option<i64>>()?);
}
"watermark" => {
if f_watermark.is_some() {
return Err(::serde::de::Error::duplicate_field("watermark"));
}
f_watermark = Some(map.next_value::<Option<WatermarkRequestConfig>>()?);
}
other => {
return Err(::serde::de::Error::unknown_field(other, FIELDS));
}
@@ -1416,6 +1427,76 @@ impl<'de> ::serde::Deserialize<'de> for SamplingParams {
sampling_seed: f_sampling_seed.flatten(),
custom_params: f_custom_params.flatten(),
beam_width: f_beam_width.flatten(),
watermark: f_watermark.flatten(),
})
}
}
deserializer.deserialize_map(V)
}
}
impl ::serde::Serialize for WatermarkRequestConfig {
fn serialize<S: ::serde::Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
let mut map = serializer.serialize_map(None)?;
if let Some(v) = &self.enabled {
::serde::ser::SerializeMap::serialize_entry(&mut map, "enabled", v)?;
}
if let Some(v) = &self.key {
::serde::ser::SerializeMap::serialize_entry(&mut map, "key", v)?;
}
if let Some(v) = &self.context_window {
::serde::ser::SerializeMap::serialize_entry(&mut map, "context_window", v)?;
}
::serde::ser::SerializeMap::end(map)
}
}
impl<'de> ::serde::Deserialize<'de> for WatermarkRequestConfig {
fn deserialize<D: ::serde::Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
const FIELDS: &[&str] = &["enabled", "key", "context_window"];
struct V;
impl<'de> ::serde::de::Visitor<'de> for V {
type Value = WatermarkRequestConfig;
fn expecting(&self, f: &mut ::core::fmt::Formatter) -> ::core::fmt::Result {
f.write_str("struct WatermarkRequestConfig")
}
fn visit_map<A: ::serde::de::MapAccess<'de>>(
self,
mut map: A,
) -> Result<Self::Value, A::Error> {
let mut f_enabled = None;
let mut f_key = None;
let mut f_context_window = None;
while let Some(key) = map.next_key::<::std::borrow::Cow<'_, str>>()? {
match key.as_ref() {
"enabled" => {
if f_enabled.is_some() {
return Err(::serde::de::Error::duplicate_field("enabled"));
}
f_enabled = Some(map.next_value::<Option<bool>>()?);
}
"key" => {
if f_key.is_some() {
return Err(::serde::de::Error::duplicate_field("key"));
}
f_key =
Some(map.next_value::<Option<::prost::alloc::string::String>>()?);
}
"context_window" => {
if f_context_window.is_some() {
return Err(::serde::de::Error::duplicate_field("context_window"));
}
f_context_window = Some(map.next_value::<Option<i64>>()?);
}
other => {
return Err(::serde::de::Error::unknown_field(other, FIELDS));
}
}
}
Ok(WatermarkRequestConfig {
enabled: f_enabled.flatten(),
key: f_key.flatten(),
context_window: f_context_window.flatten(),
})
}
}
@@ -4,6 +4,7 @@
//! `TokenizerManager._create_tokenized_object` does).
use std::collections::BTreeMap;
use std::fmt;
use serde::{Deserialize, Serialize};
@@ -90,6 +91,27 @@ fn max_new_tokens_default() -> Option<i64> {
Some(128)
}
#[derive(Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct WatermarkRequestConfig {
#[serde(default)]
pub enabled: Option<bool>,
#[serde(default)]
pub key: Option<String>,
#[serde(default)]
pub context_window: Option<i64>,
}
impl fmt::Debug for WatermarkRequestConfig {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("WatermarkRequestConfig")
.field("enabled", &self.enabled)
.field("key", &"<redacted>")
.field("context_window", &self.context_window)
.finish()
}
}
/// The sampling parameters of one `/generate` request. Deserialized from the
/// client's `sampling_params` object (unknown keys are a 400, mirroring Python's
/// `SamplingParams(**kwargs)` TypeError) and serialized by field name into the
@@ -209,6 +231,8 @@ pub struct SamplingParams {
/// Python's `CustomParamValue` exactly.
#[serde(default)]
pub custom_params: Option<BTreeMap<String, CustomParamValue>>,
#[serde(default)]
pub watermark: Option<WatermarkRequestConfig>,
// Normalized internal fields.
//
@@ -270,6 +294,7 @@ impl Default for SamplingParams {
spaces_between_special_tokens: bool_true::default(),
no_stop_trim: bool_false::default(),
custom_params: None,
watermark: None,
stream_interval: None,
logit_bias: None,
sampling_seed: None,
@@ -724,6 +749,7 @@ mod tests {
"logit_bias",
"sampling_seed",
"custom_params",
"watermark",
"stop_strs",
"stop_regex_strs",
"stop_str_max_len",
@@ -803,12 +829,30 @@ mod tests {
assert!(arr[at("stop")].is_nil());
assert!(arr[at("logit_bias")].is_nil());
assert!(arr[at("custom_params")].is_nil());
assert!(arr[at("watermark")].is_nil());
// `normalize` outputs occupy the tail.
assert!(arr[at("stop_strs")].is_array());
assert_eq!(arr[at("is_normalized")].as_bool(), Some(false));
assert_eq!(arr[at("ebnf_full_assistant")].as_bool(), Some(false));
}
#[test]
fn watermark_key_is_typed_and_debug_redacted() {
let key = "0123456789abcdef";
let sampling = norm(
&serde_json::json!({
"watermark": {"enabled": true, "key": key, "context_window": 4}
})
.to_string(),
);
let watermark = sampling.watermark.as_ref().expect("watermark parsed");
assert_eq!(watermark.key.as_deref(), Some(key));
assert_eq!(watermark.context_window, Some(4));
assert_eq!(watermark.enabled, Some(true));
assert!(!format!("{sampling:?}").contains(key));
}
#[test]
fn verify_rejects_out_of_range() {
for (json, want) in [
+21 -1
View File
@@ -11,7 +11,7 @@ use std::collections::BTreeMap;
use sglang_api_types::api::v1 as api;
use super::multimodal::{MediaHints, MmDataInput, MmItem};
use super::sampling::{CustomParamValue, SamplingParams};
use super::sampling::{CustomParamValue, SamplingParams, WatermarkRequestConfig};
use super::types::{OneOrMany, TokenIds};
use crate::utils::error::Error;
@@ -165,6 +165,11 @@ impl TryFrom<api::SamplingParams> for SamplingParams {
logit_bias: (!p.logit_bias.is_empty()).then(|| p.logit_bias.into_iter().collect()),
sampling_seed: p.sampling_seed,
custom_params,
watermark: p.watermark.map(|watermark| WatermarkRequestConfig {
enabled: watermark.enabled,
key: watermark.key,
context_window: watermark.context_window,
}),
..defaults
})
}
@@ -386,6 +391,21 @@ mod tests {
assert_eq!(p.temperature, 1.0, "schema default applied by the decoder");
}
#[test]
fn watermark_crosses_generated_schema() {
let key = "0123456789abcdef";
let p: api::SamplingParams = serde_json::from_value(serde_json::json!({
"watermark": {"enabled": true, "key": key, "context_window": 4}
}))
.unwrap();
let p = SamplingParams::try_from(p).unwrap();
let watermark = p.watermark.as_ref().unwrap();
assert_eq!(watermark.enabled, Some(true));
assert_eq!(watermark.key.as_deref(), Some(key));
assert_eq!(watermark.context_window, Some(4));
assert!(!format!("{p:?}").contains(key));
}
/// The protobuf entry fills the same keys the JSON entry does: a set field
/// wins, an unset one takes the preferred value, and an absent carrier
/// takes them all.
@@ -0,0 +1,240 @@
import json
import os
import sys
import tempfile
import pytest
import requests
from sglang.srt.sampling.watermarking import WatermarkDetector
from sglang.srt.utils import kill_process_tree
from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.test_utils import (
DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
DEFAULT_URL_FOR_TEST,
CustomTestCase,
popen_launch_server,
)
register_cuda_ci(est_time=320, stage="base-b", runner_config="1-gpu-small")
_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
_KEY_A = "0123456789abcdef"
_KEY_B = "fedcba9876543210"
_OMITTED = object()
def _watermark_statistics(choice, key):
return (
WatermarkDetector(key)
.detect_tokens(
choice["response_token_ids"],
prompt_token_ids=choice["prompt_token_ids"],
)
.combined
)
def _chat_payload(watermark=_OMITTED, *, max_tokens):
payload = {
"model": _MODEL,
"messages": [
{
"role": "user",
"content": (
"Write a long, varied field guide to urban trees. Include many "
"species, observations, and practical examples."
),
}
],
"temperature": 1.0,
"top_p": 0.95,
"max_tokens": max_tokens,
"ignore_eos": True,
"return_token_ids": True,
}
if watermark is not _OMITTED:
payload["watermark"] = watermark
return payload
def _assert_detected(response, key, *, other_key=None):
assert response.status_code == 200, response.text
choice = response.json()["choices"][0]
statistics = _watermark_statistics(choice, key)
assert statistics.num_contexts >= 100
assert statistics.z_score >= 5.0
if other_key is not None:
other = _watermark_statistics(choice, other_key)
assert statistics.z_score - other.z_score >= 4.0
class WatermarkServerTest(CustomTestCase):
config_fields = {}
process = None
config_file = None
@classmethod
def setUpClass(cls):
cls.config_file = tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False
)
json.dump(
{"key": _KEY_A, "context_window": 4, **cls.config_fields}, cls.config_file
)
cls.config_file.close()
os.chmod(cls.config_file.name, 0o600)
cls.process = popen_launch_server(
_MODEL,
DEFAULT_URL_FOR_TEST,
timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
other_args=[
"--enable-watermark",
"--watermark-config",
cls.config_file.name,
],
)
@classmethod
def tearDownClass(cls):
if cls.process is not None:
kill_process_tree(cls.process.pid)
if cls.config_file is not None:
os.unlink(cls.config_file.name)
class TestWatermarkRequestEndpoint(WatermarkServerTest):
def test_bad_request_key_is_rejected_without_echo(self):
bad_key = "not-a-hex-key"
response = requests.post(
f"{DEFAULT_URL_FOR_TEST}/v1/chat/completions",
json=_chat_payload({"key": bad_key}, max_tokens=1),
timeout=60,
)
assert response.status_code == 400
assert bad_key not in response.text
def test_per_request_keys_are_isolated(self):
for key, watermark in (
(_KEY_A, {"enabled": True}),
(_KEY_B, {"key": _KEY_B}),
):
response = requests.post(
f"{DEFAULT_URL_FOR_TEST}/v1/chat/completions",
json=_chat_payload(watermark, max_tokens=512),
timeout=180,
)
_assert_detected(
response,
key,
other_key=_KEY_B if key == _KEY_A else _KEY_A,
)
class TestWatermarkDefaultEnabledEndpoint(WatermarkServerTest):
config_fields = {"default_enabled": True}
def test_omitted_request_uses_server_key_and_opt_out_is_allowed(self):
response = requests.post(
f"{DEFAULT_URL_FOR_TEST}/v1/chat/completions",
json=_chat_payload(max_tokens=512),
timeout=180,
)
_assert_detected(response, _KEY_A, other_key=_KEY_B)
disabled = requests.post(
f"{DEFAULT_URL_FOR_TEST}/v1/chat/completions",
json=_chat_payload({"enabled": False}, max_tokens=1),
timeout=60,
)
assert disabled.status_code == 200, disabled.text
def test_structured_output_stays_valid_and_detectable(self):
schema = {
"type": "object",
"properties": {
"entries": {
"type": "array",
"minItems": 8,
"maxItems": 8,
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 4,
"maxLength": 24,
},
"habitat": {
"type": "string",
"minLength": 4,
"maxLength": 32,
},
"observation": {
"type": "string",
"minLength": 8,
"maxLength": 48,
},
"evergreen": {"type": "boolean"},
},
"required": [
"name",
"habitat",
"observation",
"evergreen",
],
"additionalProperties": False,
},
}
},
"required": ["entries"],
"additionalProperties": False,
}
payload = _chat_payload(max_tokens=768)
payload["messages"][0]["content"] = (
"Return a field guide to eight distinct trees as the requested JSON."
)
payload["ignore_eos"] = False
payload["response_format"] = {
"type": "json_schema",
"json_schema": {
"name": "tree_guide",
"strict": True,
"schema": schema,
},
}
response = requests.post(
f"{DEFAULT_URL_FOR_TEST}/v1/chat/completions",
json=payload,
timeout=180,
)
assert response.status_code == 200, response.text
choice = response.json()["choices"][0]
assert len(json.loads(choice["message"]["content"])["entries"]) == 8
statistics = _watermark_statistics(choice, _KEY_A)
assert statistics.num_contexts >= 100
assert statistics.z_score >= 3.0
class TestWatermarkEnforceAllEndpoint(WatermarkServerTest):
config_fields = {"enforce_all": True}
def test_omitted_request_uses_server_key_and_opt_out_is_rejected(self):
response = requests.post(
f"{DEFAULT_URL_FOR_TEST}/v1/chat/completions",
json=_chat_payload(max_tokens=512),
timeout=180,
)
_assert_detected(response, _KEY_A, other_key=_KEY_B)
disabled = requests.post(
f"{DEFAULT_URL_FOR_TEST}/v1/chat/completions",
json=_chat_payload({"enabled": False}, max_tokens=1),
timeout=60,
)
assert disabled.status_code == 400
if __name__ == "__main__":
sys.exit(pytest.main([__file__]))
@@ -0,0 +1,417 @@
import sys
import pytest
import torch
from sglang.kernels.ops.sampling.textseal_selector import (
can_use_finite_topk_watermark,
force_watermark_tokens_triton,
prepare_watermark_contexts_triton,
select_watermark_tokens_triton,
)
from sglang.srt.sampling.watermarking.core import (
WatermarkState,
_hash_contexts,
_truncate_probabilities,
_watermark_hash32_torch,
select_watermark_tokens_torch,
)
from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large")
def test_selector_matches_torch_across_split_boundaries():
generator = torch.Generator().manual_seed(7)
batch_size = 5
vocab_size = 16397
probabilities = torch.rand(batch_size, vocab_size, generator=generator)
support_lengths = torch.tensor([1, 8191, 8192, 8193, vocab_size])
support = torch.arange(vocab_size).view(1, -1) < support_lengths.view(-1, 1)
probabilities = torch.where(support, probabilities, 0.0)
probabilities /= probabilities.sum(dim=-1, keepdim=True)
context_hashes = torch.tensor([0, 1, 2**31 - 1, 2**32 - 1, 1145416960])
keys = torch.tensor([0, 1, -1, -(2**63), 0x0123456789ABCDEF])
expected = select_watermark_tokens_torch(probabilities, context_hashes, keys)
actual = select_watermark_tokens_triton(
probabilities.cuda(), context_hashes.cuda(), keys.cuda()
).cpu()
torch.testing.assert_close(actual.to(torch.int64), expected)
keys_b = torch.tensor([1, -1, 0, 0x1111222233334444, -(2**63)])
mixing_thresholds = torch.tensor(
[1, 1 << 30, 1 << 31, 3 << 30, (1 << 32) - 1], dtype=torch.int64
)
expected_dual = select_watermark_tokens_torch(
probabilities, context_hashes, keys, keys_b, mixing_thresholds
)
actual_dual = select_watermark_tokens_triton(
probabilities.cuda(),
context_hashes.cuda(),
keys.cuda(),
keys_b.cuda(),
mixing_thresholds.cuda(),
).cpu()
torch.testing.assert_close(actual_dual.to(torch.int64), expected_dual)
def test_hash_near_uint32_max_does_not_override_probability():
vocab_size = 149050
probabilities = torch.zeros((1, vocab_size), dtype=torch.float32)
probabilities[0, 0] = 1.0 - 1e-9
probabilities[0, 149049] = 1e-9
context_hashes = torch.tensor([704711416], dtype=torch.int64)
keys = torch.tensor([120], dtype=torch.int64)
boundary_hash = _watermark_hash32_torch(
keys, context_hashes, torch.tensor([149049])
).item()
assert boundary_hash >= (1 << 32) - 128
assert (
select_watermark_tokens_torch(probabilities, context_hashes, keys).item() == 0
)
assert (
select_watermark_tokens_triton(
probabilities.cuda(), context_hashes.cuda(), keys.cuda()
).item()
== 0
)
logits = probabilities.log().cuda()
expected_logits = torch.full_like(logits, -torch.inf)
expected_logits[0, 0] = 0.0
temperatures = torch.ones((1, 1), device="cuda")
top_ks = torch.tensor([2], dtype=torch.int32, device="cuda")
top_ps = torch.ones(1, device="cuda")
min_ps = torch.zeros(1, device="cuda")
eligible = torch.ones(1, dtype=torch.bool, device="cuda")
for max_top_k in (2, None):
actual_logits = logits.clone()
selected = force_watermark_tokens_triton(
actual_logits,
context_hashes.cuda(),
eligible,
temperatures,
top_ks,
top_ps,
min_ps,
keys.cuda(),
max_top_k=max_top_k,
)
assert selected.item() == 0
assert torch.equal(actual_logits, expected_logits)
@pytest.mark.parametrize(
("dtype", "vocab_size", "dual_key"),
[
(torch.bfloat16, 8192, False),
(torch.bfloat16, 8193, False),
(torch.bfloat16, 151936, False),
(torch.float32, 16397, False),
(torch.float16, 16397, True),
],
)
def test_fused_force_matches_torch_truncation(dtype, vocab_size, dual_key):
generator = torch.Generator(device="cuda").manual_seed(7)
logits = torch.randn((5, vocab_size), generator=generator, device="cuda").to(dtype)
temperatures = torch.tensor([[0.7], [1.0], [1.3], [0.9], [1.1]], device="cuda")
top_ks = torch.tensor(
[1, 7, min(100, vocab_size), vocab_size, vocab_size],
dtype=torch.int32,
device="cuda",
)
top_ps = torch.tensor([1.0, 0.95, 0.5, 0.01, 1.0], device="cuda")
min_ps = torch.tensor([0.0, 0.0, 0.05, 0.0, 0.2], device="cuda")
context_hashes = torch.tensor(
[0, 1, 2**31 - 1, 2**32 - 1, 1145416960],
dtype=torch.int64,
device="cuda",
)
keys = torch.tensor(
[0, 1, -1, -(2**63), 0x0123456789ABCDEF],
dtype=torch.int64,
device="cuda",
)
keys_b = torch.tensor(
[1, -1, 0, 0x1111222233334444, -(2**63)],
dtype=torch.int64,
device="cuda",
)
mixing_thresholds = torch.tensor(
[1, 1 << 30, 1 << 31, 3 << 30, (1 << 32) - 1],
dtype=torch.int64,
device="cuda",
)
eligible = torch.tensor([False, True, True, True, True], device="cuda")
expected_logits = logits.clone()
probabilities = _truncate_probabilities(
expected_logits, temperatures, top_ks, top_ps, min_ps
)
rows = eligible.nonzero(as_tuple=True)[0]
expected = torch.full((5,), -1, dtype=torch.int64, device="cuda")
expected[rows] = select_watermark_tokens_torch(
probabilities[rows].float(),
context_hashes[rows],
keys[rows],
keys_b[rows] if dual_key else None,
mixing_thresholds[rows] if dual_key else None,
)
expected_logits[rows] = -torch.inf
expected_logits[rows, expected[rows]] = 0.0
actual_logits = logits.clone()
actual = force_watermark_tokens_triton(
actual_logits,
context_hashes,
eligible,
temperatures,
top_ks,
top_ps,
min_ps,
keys,
keys_b if dual_key else None,
mixing_thresholds if dual_key else None,
)
assert torch.equal(actual.to(torch.int64), expected)
assert torch.equal(actual_logits, expected_logits)
@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float32])
@pytest.mark.parametrize("max_probability", [0.7, 1.0])
def test_finite_top_k_fast_path_matches_sort_with_ties(dtype, max_probability):
vocab_size = 8193
token_ids = torch.arange(vocab_size, device="cuda", dtype=torch.float32)
logits = torch.stack(
(
torch.zeros_like(token_ids),
token_ids.remainder(5),
torch.round(torch.sin(token_ids) * 8),
torch.where(token_ids < 5, torch.ones_like(token_ids), -torch.inf),
)
).to(dtype)
temperatures = torch.tensor([[0.7], [1.0], [1.3], [0.9]], device="cuda")
top_ks = torch.tensor([2, 7, 20, 64], dtype=torch.int32, device="cuda")
top_ps = torch.tensor([1.0, 0.95, 0.5, 1.0], device="cuda")
min_ps = torch.tensor([0.0, 0.0, 0.05, 0.2], device="cuda")
context_hashes = torch.tensor(
[0, 1, 2**32 - 1, 1145416960], dtype=torch.int64, device="cuda"
)
keys = torch.tensor(
[0, 1, -1, 0x0123456789ABCDEF], dtype=torch.int64, device="cuda"
)
eligible = torch.ones(4, dtype=torch.bool, device="cuda")
sort_logits = logits.clone()
sort_selected = force_watermark_tokens_triton(
sort_logits,
context_hashes,
eligible,
temperatures,
top_ks,
top_ps,
min_ps,
keys,
max_probability=max_probability,
)
fast_logits = logits.clone()
fast_selected = force_watermark_tokens_triton(
fast_logits,
context_hashes,
eligible,
temperatures,
top_ks,
top_ps,
min_ps,
keys,
max_top_k=64,
max_probability=max_probability,
)
assert torch.equal(fast_selected, sort_selected)
assert torch.equal(fast_logits, sort_logits)
keys_b = torch.tensor(
[1, -1, 0x1111222233334444, -(2**63)], dtype=torch.int64, device="cuda"
)
mixing_thresholds = torch.tensor(
[1, 1 << 30, 1 << 31, (1 << 32) - 1],
dtype=torch.int64,
device="cuda",
)
dual_sort_logits = logits.clone()
dual_sort_selected = force_watermark_tokens_triton(
dual_sort_logits,
context_hashes,
eligible,
temperatures,
top_ks,
top_ps,
min_ps,
keys,
keys_b,
mixing_thresholds,
max_probability=max_probability,
)
dual_fast_logits = logits.clone()
dual_fast_selected = force_watermark_tokens_triton(
dual_fast_logits,
context_hashes,
eligible,
temperatures,
top_ks,
top_ps,
min_ps,
keys,
keys_b,
mixing_thresholds,
max_top_k=64,
max_probability=max_probability,
)
assert torch.equal(dual_fast_selected, dual_sort_selected)
assert torch.equal(dual_fast_logits, dual_sort_logits)
@pytest.mark.parametrize("max_top_k", [8191, 8192])
def test_finite_top_k_boundary_matches_sort_with_ties(max_top_k):
vocab_size = 8193
token_ids = torch.arange(vocab_size, device="cuda", dtype=torch.float32)
logits = torch.stack((torch.zeros_like(token_ids), token_ids.remainder(7))).to(
torch.bfloat16
)
temperatures = torch.tensor([[0.7], [1.3]], device="cuda")
top_ks = torch.full((2,), max_top_k, dtype=torch.int32, device="cuda")
top_ps = torch.tensor([0.95, 0.5], device="cuda")
min_ps = torch.tensor([0.0, 0.05], device="cuda")
context_hashes = torch.tensor([1, 1145416960], dtype=torch.int64, device="cuda")
keys = torch.tensor([1, 0x0123456789ABCDEF], dtype=torch.int64, device="cuda")
eligible = torch.ones(2, dtype=torch.bool, device="cuda")
assert can_use_finite_topk_watermark(max_top_k, vocab_size)
assert not can_use_finite_topk_watermark(8193, 8194)
sort_logits = logits.clone()
sort_selected = force_watermark_tokens_triton(
sort_logits,
context_hashes,
eligible,
temperatures,
top_ks,
top_ps,
min_ps,
keys,
)
fast_logits = logits.clone()
fast_selected = force_watermark_tokens_triton(
fast_logits,
context_hashes,
eligible,
temperatures,
top_ks,
top_ps,
min_ps,
keys,
max_top_k=max_top_k,
)
assert torch.equal(fast_selected, sort_selected)
assert torch.equal(fast_logits, sort_logits)
def test_fused_context_state_matches_torch():
max_contexts = 2048
reference = WatermarkState(
max_num_reqs=8,
context_window=4,
max_contexts_per_req=max_contexts,
key="0123456789abcdef",
device="cuda",
)
actual = WatermarkState(
max_num_reqs=8,
context_window=4,
max_contexts_per_req=max_contexts,
key="0123456789abcdef",
device="cuda",
)
req_pool_indices = torch.tensor([1, 3, 5, 7], dtype=torch.int32, device="cuda")
tails = [[10, 11, 12, 13], [20, 21], [30, 31, 32, 33], [40, 41, 42]]
reference.init_from_prompt(req_pool_indices, tails)
actual.init_from_prompt(req_pool_indices, tails)
counts = torch.tensor([0, 1, 1025, 1500], dtype=torch.int32, device="cuda")
reference.num_watermarked_contexts[req_pool_indices.long()] = counts
actual.num_watermarked_contexts.copy_(reference.num_watermarked_contexts)
generator = torch.Generator(device="cuda").manual_seed(9)
history = torch.randint(
-(2**31),
2**31 - 1,
reference.watermarked_context_hashes.shape,
dtype=torch.int32,
device="cuda",
generator=generator,
)
reference.watermarked_context_hashes.copy_(history)
actual.watermarked_context_hashes.copy_(history)
context_windows = torch.tensor([4, 1, 3, 2], dtype=torch.int32, device="cuda")
watermark_enabled = torch.tensor([True, True, True, True], device="cuda")
top_ks = torch.tensor([64, 64, 64, 1], dtype=torch.int32, device="cuda")
contexts, context_lengths = reference.contexts_tail(
req_pool_indices, context_windows
)
expected_hashes = _hash_contexts(contexts, context_lengths)
reference.watermarked_context_hashes[req_pool_indices[1], 0] = expected_hashes[
1
].to(torch.int32)
reference.watermarked_context_hashes[req_pool_indices[2], 1024] = expected_hashes[
2
].to(torch.int32)
actual.watermarked_context_hashes.copy_(reference.watermarked_context_hashes)
initial_counts = reference.num_watermarked_contexts.clone()
expected_eligible = reference._new_context_mask(
req_pool_indices,
expected_hashes,
watermark_enabled & (top_ks > 1) & (context_lengths > 0),
)
reference._record_contexts(req_pool_indices, expected_hashes, expected_eligible)
actual_hashes = torch.empty(4, dtype=torch.int64, device="cuda")
actual_eligible = torch.empty(4, dtype=torch.bool, device="cuda")
prepare_watermark_contexts_triton(
actual.token_ids,
actual.lengths,
actual.write_positions,
actual.watermarked_context_hashes,
actual.num_watermarked_contexts,
req_pool_indices,
context_windows,
watermark_enabled,
top_ks,
actual_hashes,
actual_eligible,
)
torch.testing.assert_close(actual_hashes, expected_hashes)
torch.testing.assert_close(actual_eligible, expected_eligible)
torch.testing.assert_close(
actual.num_watermarked_contexts, reference.num_watermarked_contexts
)
for row, pool_index in enumerate(req_pool_indices.tolist()):
if expected_eligible[row]:
position = initial_counts[pool_index]
assert (
actual.watermarked_context_hashes[pool_index, position]
== reference.watermarked_context_hashes[pool_index, position]
)
if __name__ == "__main__":
sys.exit(pytest.main([__file__]))
@@ -0,0 +1,320 @@
import sys
from types import SimpleNamespace
from unittest.mock import Mock
import pytest
import torch
from sglang.srt.sampling.watermarking.core import (
WatermarkState,
normalize_watermark_request,
)
from sglang.srt.sampling.watermarking.detector import hash_context
from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=10, stage="base-b-kernel-unit", runner_config="1-gpu-large")
def test_repeated_context_and_greedy_bypass():
device = "cuda"
state = WatermarkState(
max_num_reqs=2,
context_window=2,
max_contexts_per_req=8,
key="0123456789abcdef",
device=device,
default_enabled=True,
)
req_pool_indices = torch.tensor([0, 1], device=device, dtype=torch.int32)
state.init_from_prompt(req_pool_indices, [[10, 11], [20, 21]])
sampling_info = SimpleNamespace(
temperatures=torch.ones((2, 1), device=device),
top_ks=torch.tensor([64, 1], device=device, dtype=torch.int32),
top_ps=torch.ones(2, device=device),
min_ps=torch.zeros(2, device=device),
max_top_k=64,
watermark_keys=None,
watermark_context_windows=None,
watermark_enabled=None,
has_watermark_candidates=True,
)
first_logits = torch.zeros((2, 64), device=device)
state.force(first_logits, req_pool_indices, sampling_info)
assert torch.isfinite(first_logits[0]).sum().item() == 1
assert torch.equal(first_logits[1], torch.zeros(64, device=device))
assert state.num_watermarked_contexts.tolist() == [1, 0]
state.append(
req_pool_indices[:1], torch.tensor([10], device=device, dtype=torch.int32)
)
state.append(
req_pool_indices[:1], torch.tensor([11], device=device, dtype=torch.int32)
)
repeated_logits = torch.zeros((2, 64), device=device)
state.force(repeated_logits, req_pool_indices, sampling_info)
assert torch.equal(repeated_logits, torch.zeros_like(repeated_logits))
assert state.num_watermarked_contexts.tolist() == [1, 0]
def test_low_entropy_bypass_does_not_consume_context():
state = WatermarkState(
max_num_reqs=1,
context_window=2,
max_contexts_per_req=8,
key="0123456789abcdef",
device="cuda",
default_enabled=True,
max_probability=0.5,
)
req_pool_indices = torch.tensor([0], device="cuda", dtype=torch.int32)
state.init_from_prompt(req_pool_indices, [[10, 11]])
sampling_info = SimpleNamespace(
temperatures=torch.ones((1, 1), device="cuda"),
top_ks=torch.tensor([2], device="cuda", dtype=torch.int32),
top_ps=torch.ones(1, device="cuda"),
min_ps=torch.zeros(1, device="cuda"),
max_top_k=2,
watermark_keys=None,
watermark_context_windows=None,
watermark_enabled=None,
has_watermark_candidates=True,
)
low_entropy_logits = torch.full((1, 64), -torch.inf, device="cuda")
low_entropy_logits[0, :2] = torch.tensor([1.0, 0.0], device="cuda")
original = low_entropy_logits.clone()
state.force(low_entropy_logits, req_pool_indices, sampling_info)
assert torch.equal(low_entropy_logits, original)
assert state.num_watermarked_contexts[0].item() == 0
boundary_logits = torch.full((1, 64), -torch.inf, device="cuda")
boundary_logits[0, :2] = 0
state.force(boundary_logits, req_pool_indices, sampling_info)
assert torch.isfinite(boundary_logits).sum().item() == 1
assert state.num_watermarked_contexts[0].item() == 1
def test_inactive_batch_skips_watermark_state(monkeypatch):
state = WatermarkState(
max_num_reqs=1,
context_window=2,
max_contexts_per_req=8,
key="0123456789abcdef",
device="cuda",
default_enabled=True,
)
req_pool_indices = torch.tensor([0], device="cuda", dtype=torch.int32)
kernel_activity = Mock()
monkeypatch.setattr(state, "_ensure_selection_buffers", kernel_activity)
sampling_info = SimpleNamespace(
temperatures=torch.ones((1, 1), device="cuda"),
top_ks=torch.tensor([64], device="cuda", dtype=torch.int32),
top_ps=torch.ones(1, device="cuda"),
min_ps=torch.zeros(1, device="cuda"),
max_top_k=64,
watermark_keys=None,
watermark_context_windows=None,
watermark_enabled=torch.tensor([False], device="cuda"),
has_watermark_candidates=False,
)
state.force(torch.zeros((1, 64), device="cuda"), req_pool_indices, sampling_info)
kernel_activity.assert_not_called()
assert state.lengths[0].item() == 0
def test_retracted_request_restores_context_history():
device = "cuda"
state = WatermarkState(
max_num_reqs=2,
context_window=2,
max_contexts_per_req=8,
key="0123456789abcdef",
device=device,
default_enabled=True,
)
req_pool_indices = torch.tensor([1], device=device, dtype=torch.int32)
request = SimpleNamespace(
retracted_stain=True,
origin_input_ids=[10, 11],
output_ids=[10, 11],
sampling_params=SimpleNamespace(
watermark=normalize_watermark_request(
{"enabled": True, "context_window": 2}
),
top_k=64,
),
get_fill_ids=lambda: [10, 11, 10, 11],
)
forward_mode = SimpleNamespace(
is_extend_without_speculative=lambda: True,
is_mixed=lambda: False,
)
batch = SimpleNamespace(
forward_mode=forward_mode,
reqs=[request],
decoding_reqs=None,
)
history = state.retracted_context_hashes(batch)
assert history is not None
expected_hashes = {
hash_context([10, 11]),
hash_context([11, 10]),
}
assert {
value if value >= 0 else value + 2**32 for value in history[0]
} == expected_hashes
state.init_from_prompt(
req_pool_indices,
state.prompt_tails(batch),
history,
)
sampling_info = SimpleNamespace(
temperatures=torch.ones((1, 1), device=device),
top_ks=torch.tensor([64], device=device, dtype=torch.int32),
top_ps=torch.ones(1, device=device),
min_ps=torch.zeros(1, device=device),
max_top_k=64,
watermark_keys=None,
watermark_context_windows=None,
watermark_enabled=None,
has_watermark_candidates=True,
)
logits = torch.zeros((1, 64), device=device)
state.force(logits, req_pool_indices, sampling_info)
assert torch.equal(logits, torch.zeros_like(logits))
assert state.num_watermarked_contexts[1].item() == 2
def test_speculative_record_stops_at_context_capacity():
state = WatermarkState(
max_num_reqs=1,
context_window=2,
max_contexts_per_req=2,
key="0123456789abcdef",
device="cuda",
default_enabled=True,
)
state.num_watermarked_contexts[0] = 1
state.watermarked_context_hashes[0, 0] = 10
state.record_speculative(
torch.tensor([0], dtype=torch.int32, device="cuda"),
torch.tensor([20, 30, 40], dtype=torch.int64, device="cuda"),
torch.ones(3, dtype=torch.bool, device="cuda"),
torch.tensor([[0, 1, 2]], dtype=torch.int64, device="cuda"),
torch.tensor([3], dtype=torch.int32, device="cuda"),
)
assert state.num_watermarked_contexts[0].item() == 2
assert state.watermarked_context_hashes[0].tolist() == [10, 20]
def test_speculative_tree_fast_path_matches_tensor_reference():
draft_token_num = 4
req_pool_indices = torch.tensor([1, 3], dtype=torch.int32, device="cuda")
prompt_tails = [[10, 11, 12, 13], [20, 21, 22, 23]]
states = [
WatermarkState(
max_num_reqs=4,
context_window=4,
max_contexts_per_req=16,
key="0123456789abcdef",
key_b="fedcba9876543210",
mixing_probability=0.5,
device="cuda",
)
for _ in range(2)
]
for state in states:
state.init_from_prompt(req_pool_indices, prompt_tails)
draft_tokens = torch.tensor(
[[30, 31, 31, 33], [40, 41, 41, 43]],
dtype=torch.int32,
device="cuda",
).flatten()
positions = torch.tensor(
[[6, 7, 7, 8], [4, 5, 5, 6]], dtype=torch.int64, device="cuda"
).flatten()
tree_rows = (
(True, False, False, False),
(True, True, False, False),
(True, False, True, False),
(True, True, False, True),
)
custom_mask = torch.tensor(
[
value
for prefix_length in (6, 4)
for row in tree_rows
for value in (True,) * prefix_length + row
],
dtype=torch.bool,
device="cuda",
)
sampling_info = SimpleNamespace(
temperatures=torch.tensor([[0.7], [1.3]], device="cuda"),
top_ks=torch.tensor([20, 31], dtype=torch.int32, device="cuda"),
top_ps=torch.tensor([0.95, 0.8], device="cuda"),
min_ps=torch.tensor([0.0, 0.05], device="cuda"),
max_top_k=31,
watermark_keys=torch.tensor(
[0x0123456789ABCDEF, 0x1111222233334444],
dtype=torch.int64,
device="cuda",
),
watermark_context_windows=torch.tensor(
[4, 3], dtype=torch.int32, device="cuda"
),
watermark_enabled=torch.ones(2, dtype=torch.bool, device="cuda"),
)
logits = torch.randn(
(8, 257), generator=torch.Generator("cuda").manual_seed(9), device="cuda"
)
reference_logits = logits.clone()
contexts, context_lengths = states[0].speculative_contexts(
req_pool_indices,
draft_tokens,
custom_mask,
positions,
draft_token_num,
True,
sampling_info.watermark_context_windows,
)
reference_hashes, reference_selected = states[0].force_speculative(
reference_logits,
req_pool_indices,
contexts,
context_lengths,
sampling_info,
draft_token_num,
)
fused_logits = logits.clone()
fused_hashes, fused_selected = states[1].force_speculative_from_tree(
fused_logits,
req_pool_indices,
draft_tokens,
custom_mask,
positions,
sampling_info,
draft_token_num,
True,
)
assert torch.equal(fused_hashes, reference_hashes)
assert torch.equal(fused_selected, reference_selected)
assert torch.equal(fused_logits, reference_logits)
assert fused_selected.view(2, draft_token_num)[:, 1].all()
assert fused_selected.view(2, draft_token_num)[:, 2].logical_not().all()
if __name__ == "__main__":
sys.exit(pytest.main([__file__]))
@@ -89,6 +89,7 @@ PYTHON_ONLY_REQUEST_FIELDS = frozenset(
"use_audio_in_video",
"video_config",
"video_max_dynamic_patch",
"watermark",
}
)
@@ -214,6 +214,13 @@ class TestModelOverridableWhitelist(CustomTestCase):
"enable_symm_mem",
"speculative_attention_mode",
"speculative_draft_attention_backend",
"watermark_context_window",
"watermark_default_enabled",
"watermark_enforce_all",
"watermark_key",
"watermark_key_b",
"watermark_max_probability",
"watermark_mixing_probability",
"prefill_decode_interval",
"radix_eviction_policy",
"mm_preprocess_cache_size_mb",