Скачать или смотреть Team 14 (AipAGluZ) - Indic Tokenizer

Natural Language Processing (NLP) Course (CS613) – IIT Gandhinagar

Tokenization plays a pivotal role in multilin- gual NLP. However, existing tokenizers are of- ten skewed towards high-resource languages, limiting their effectiveness for linguistically di- verse and morphologically rich languages such as those in the Indian subcontinent. This pa- per presents a comprehensive intrinsic evalu- ation of tokenization strategies across 17 In- dian languages. We quantify the trade-offs between bottom-up and top-down tokenizer algorithms (BPE and Unigram LM), effects of vocabulary sizes, and compare strategies of multilingual vocabulary construction such as joint and cluster-based training. We also show that extremely low-resource languages can benefit from tokenizers trained on related high-resource languages. Our study provides practical insights for building more fair, effi- cient, and linguistically informed tokenizers for multilingual NLP.

These presentations were created as part of the NLP course at IIT Gandhinagar.

Paper Title: Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights

Presented By:
Bhavya Aditya
Lalith Seervi
Rajul Yadav
Rutuja Swami
Sohitha Sonalika
Het Pathak
Venugopal Bhamidi

For more information, visit the course website: https://sites.google.com/iitgn.ac.in/...