Скачать или смотреть Evolution of Language Technology: From Bag of Words to Generative AI | IEEE North Jesresy

#NLP #GenerativeAI #LanguageTechnology #IEEE

Evolution of Language Technology : From Bag of Words to Geenrative AI.
Sponsored Talk under the aegis of IEEE Systems Council and Oslen College of Engineering and Science, FDU

Since 2022, the Generative AI revolution has taken the world by storm with its immense potential and promises.

At the heart of this transformation are large language models based on transformer architecture—particularly auto-regressive, pre-trained generative models like GPT. But this revolution didn’t happen overnight. It is the result of decades of iterative progress in the broader field of language technology, which consistently sought ways to make human-readable text understandable to machines through a series of increasingly sophisticated
techniques.

This presentation offers a retrospective techno-functional overview of that evolution, spotlighting the key
milestones that shaped the journey. Back in the 1970s, techniques like TF-IDF provided basic statistical
relevance scores based on word frequency. While not true embeddings, they offered foundational documentspecific
representations. A significant leap occurred around 2011 with the rise of neural language models, eventually leading to the development of global, pre-trained word embeddings like Word2Vec (2013) and GloVe
(2014), which captured semantic relationships and marked a new era.
The year 2018 stands out as an inflection point, with the arrival of BERT from Google and GPT from OpenAI— two transformative models leveraging the attention mechanism of transformers. BERT set new benchmarks in
natural language understanding, excelling at sentence-level tasks with bidirectional context. GPT, on the other hand, opened doors to generative capabilities, surprising the world with its ability to produce human-like text,
although early versions faced issues like repetitive output.

The journey of GPT since then has focused on overcoming these limitations through advancements in decoding strategies, such as top-k and nucleus sampling, significantly improving fluency and diversity in generated outputs. This talk draws on landmark research papers and modeling strategies that have defined this multi-decade
evolution, offering attendees an engaging tour of how language technology has progressed from basic keyword counting to the generative intelligence we see today.
Bag of Words | TF-IDF | Word2Vec | BERT | GPT | Natural Language Processing