minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[Spec][Ngram] 4/N: Remove max_match_window_size and min_match_window_size, matching all suffixes of the Trie (#21225)

Browse Source
This commit is contained in:
Khoa Pham
2026-04-01 22:09:46 -07:00
committed by GitHub
parent 269589ad71
commit f836658077
13 changed files with 46 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/speculative_decoding.md
+3 -8
View File
@@ -387,13 +387,11 @@ Enable it with:
| Parameter | Description | Default |
|---|---|---|
| `--speculative-num-draft-tokens` | Number of draft tokens verified per step. If omitted, defaults to `--speculative-ngram-max-match-window-size`. | `12` (with default ngram settings) |
| `--speculative-ngram-min-match-window-size` | Minimum matching window size. | `1` |
| `--speculative-ngram-max-match-window-size` | Maximum matching window size. | `12` |
| `--speculative-num-draft-tokens` | Number of draft tokens verified per step. If omitted, defaults to `min(--speculative-ngram-max-trie-depth, 12)`. | `12` (with default ngram settings) |
| `--speculative-ngram-min-bfs-breadth` | Minimum BFS breadth. | `1` |
| `--speculative-ngram-max-bfs-breadth` | Maximum BFS breadth. | `10` |
| `--speculative-ngram-match-type` | Ngram tree-building mode: `"BFS"` for recency-based expansion or `"PROB"` for frequency-based expansion. | `"BFS"` |
| `--speculative-ngram-max-trie-depth` | The max trie depth for ngram speculative decoding. | `18` |
| `--speculative-ngram-max-trie-depth` | Maximum suffix length stored and matched by the ngram trie. | `18` |
| `--speculative-ngram-capacity` | Cache capacity (number of entries). | `10,000,000` |
Notes:
@@ -408,7 +406,6 @@ python3 -m sglang.launch_server \
--model Qwen/Qwen2.5-7B-Instruct \
--speculative-algorithm NGRAM \
--speculative-num-draft-tokens 16 \
--speculative-ngram-max-match-window-size 12 \
--speculative-ngram-max-bfs-breadth 10 \
--mem-fraction-static 0.7 \
--cuda-graph-max-bs 8 \
@@ -464,12 +461,10 @@ Below is a comprehensive list of all speculative decoding parameters available i
| Parameter | Type | Default | Description |
|---|---|---|---|
| `--speculative-ngram-min-match-window-size` | `int` | `1` | Minimum ngram matching window |
| `--speculative-ngram-max-match-window-size` | `int` | `12` | Maximum ngram matching window |
| `--speculative-ngram-min-bfs-breadth` | `int` | `1` | Minimum BFS breadth |
| `--speculative-ngram-max-bfs-breadth` | `int` | `10` | Maximum BFS breadth |
| `--speculative-ngram-match-type` | `str` | `"BFS"` | Ngram tree-building mode: `"BFS"` for recency-based expansion or `"PROB"` for frequency-based expansion |
| `--speculative-ngram-max-trie-depth` | `int` | `18` | Max trie depth for ngram speculative decoding |
| `--speculative-ngram-max-trie-depth` | `int` | `18` | Maximum suffix length stored and matched by the ngram trie |
| `--speculative-ngram-capacity` | `int` | `10,000,000` | Cache capacity |
### Environment variables