minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Fix TiktokenTokenizer missing num_special_tokens_to_add (#30643)

Browse Source 
Co-authored-by: Sushil Dubey <sushildubey171@users.noreply.github.com>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
This commit is contained in:
sushil Dubey
2026-07-10 12:37:30 +00:00
committed by GitHub
co-authored by Sushil Dubey Mohammad Miadh Angkad
parent 7045e0fdff
commit 94de28764c
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/sglang/srt/tokenizer/tiktoken_tokenizer.py
+4
View File
@@ -113,6 +113,10 @@ class TiktokenTokenizer:
def decode(self, x, *args, **kwargs): def decode(self, x, *args, **kwargs):
return self.tokenizer.decode(x) return self.tokenizer.decode(x)
def num_special_tokens_to_add(self, *args, **kwargs) -> int:
# tiktoken's encode adds no special tokens (add_special_tokens is ignored).
return 0
def batch_decode( def batch_decode(
self, batch, skip_special_tokens=True, spaces_between_special_tokens=False self, batch, skip_special_tokens=True, spaces_between_special_tokens=False
): ):