Скачать или смотреть Diana Abagyan - One Tokenizer To Rule Them All Emergent Language Plasticity via Multilingual Tokeni

Join us for an insightful discussion with Diana Abagyan, author of “One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers” (https://arxiv.org/abs/2506.10766). In this session, Diana will delve into the challenges of pretraining massively multilingual Large Language Models (LLMs) and propose a simple and cost effective intervention at the start of pretraining to enhance “language plasticity”, the model’s ability to adapt to new languages, in post-training.The talk will focus on the critical role of tokenizer design in improving language adaptation capabilities. Training the tokenizer on more languages than the primary pretraining languages, or a "universal" tokenizer, enables efficient expansion of language coverage after pretraining. Through systematic experiments across diverse language groups and training strategies, she demonstrates that using a universal tokenizer significantly enhances language adaptation, achieving up to 20.2% higher win rates compared to tokenizers specific to pretraining languages. Remarkably, this adaptation is achieved with minimal compromise in performance on the majority of languages included in pretraining. This plasticity also enables adaptation in the most difficult setting, to low resource languages entirely unseen during tokenizer training and pretraining, with up to 5% win rate gains.