Скачать или смотреть Team 5 (Polyglots) - Language Identification

Natural Language Processing (NLP) Course (CS613) – IIT Gandhinagar

We create publicly available language iden- tification (LID) datasets and models in all 22 Indian languages listed in the Indian con- stitution in both native-script and romanized text. First, we create Bhasha-Abhijnaanam, a language identification test set for native- script as well as romanized text which spans all 22 Indic languages. We also train Indi- cLID, a language identifier for all the above- mentioned languages in both native and ro- manized script. For native-script text, it has better language coverage than existing LIDs and is competitive or better than other LIDs. IndicLID is the first LID for romanized text in Indian languages. Two major challenges for romanized text LID are the lack of train- ing data and low-LID performance when lan- guages are similar. We provide simple and effective solutions to these problems. In gen- eral, there has been limited work on romanized text in any language, and our findings are rel- evant to other languages that need romanized language identification. Our models are pub- licly available at https://ai4bharat.iitm. ac.in/indiclid under open-source licenses. Our training and test sets are also publicly available at https://ai4bharat.iitm.ac. in/bhasha-abhijnaanam under open-source licenses.

These presentations were created as part of the NLP course at IIT Gandhinagar.

Paper Title: Bhasha-Abhijnaanam: Native-script and Romanized Language Identification for 22 Indic Languages

Presented By:
Arka Dutta
Kancheti Aparna Mala
Irengbam Gangular Singh
Zayed Mudassir
Shubham Kumar
Smrutee Behera
Abhishek Mahor

For more information, visit the course website: https://sites.google.com/iitgn.ac.in/...