3 people 94de28764c Fix TiktokenTokenizer missing num_special_tokens_to_add (#30643) ...

Co-authored-by: Sushil Dubey <sushildubey171@users.noreply.github.com>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>

2026-07-10 12:37:30 +00:00

..

tiktoken_tokenizer.py

Fix TiktokenTokenizer missing num_special_tokens_to_add (#30643)

2026-07-10 12:37:30 +00:00