Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations

Brian Siyuan Zheng, Alisa Liu, Orevaoghene Ahia, Jonathan Hayase, Yejin Choi, Noah A. Smith
July, 2025
Paper BibTeX Code
tokenization
Alisa Liu
Alisa Liu

PhD student in natural language processing