Word Embeddings Compression for Neural Language Models | Amit Kumar Jaiswal

Welcome to the twentieth London Information Retrieval Meetup, a free evening event aimed at Information Retrieval enthusiasts and professionals who are curious to explore and discuss the latest trends in the field.

The second talk is from Amit Kumar Jaiswal, Postdoc at the University of Surrey and Honorary Research Fellow, UCL

Title: "Word Embeddings Compression for Neural Language Models"

Abstract: "Conventional neural word embeddings typically rely on a more diverse vocabulary. However, language models tend to encompass major vocabularies through word embedding parameters, particularly in multilingual models that incorporate a substantial portion of their overall learning parameters. Established techniques for compressing general neural models involve low-precision computation, quantisation, binarisation, dimensionality reduction, network pruning, SVD-based weight matrix decomposition, and knowledge distillation. In the context of neural language models, emphasis has been placed on compressing extensive word embedding matrices. Despite the prevalent use of tokenisation (SentencePiece) in practical Transformer implementations, the effectiveness of tokenisation as a form of compression. Most importantly, does it actually always guarantee the retaining the performance across varied natural language understanding task? This talk will provide you with the answers and an understanding of why and how language models compression in a meaningful way."

If you are willing to attend the next London Information Retrieval Meetup, don't forget to join our group: https://bit.ly/2IjSBeX

We are also accepting talks for the next meetups. If you have any talk you would like to propose, feel free to send us an abstract at [email protected].

**********************************
Hosted by Sease: https://sease.io