Скачать или смотреть dScience Lunch Seminar: Large language models under the hood

Welcome to our dScience lunch seminar in the Science Library, where Andrey Kutuzov will talk about ChatGPT.

In the last few years, radical increase in the scale of deep neural language models (both in terms of the size of the training data and the size of the models themselves) has led to impressive achievements in various natural language processing tasks. "Celebrity" models, like ChatGPT, LLaMa, BLOOM or PaLM are already sometimes described to as "approaching artificial intelligence", although the reality can differ from over-hyped media coverage.

In this talk, Kutuzov will describe the foundations of the technology behind large-scale language models. Two most important components behind their success are 1) state-of-the-art deep learning architectures (in particular, Transformer) and 2) the availability of tremendous amount of textual data used to train such models. The interaction of these two poses intricate theoretical and practical questions, also linked to issues with unequal distribution of computing resources. Do we have enough good-quality training data for languages other than English? Is "data poisoning" with automatically generated texts is a real danger? Why is it important to open source both training data and model weights?

Speaker
Andrey Kutuzov (PhD, UiO, 2020) is an Associate Professor in the Language Technology Group at the University of Oslo. He currently serves as the Norwegian on-site manager of the High-Performance Language Technology (HPLT) project. His academic interests include Computational linguistics and natural language processing, semantic change detection and diachronically aware language models, distributional semantics, machine learning and large-scale language models. In 2022, Kutuzov received the Norwegian Artificial Intelligence Research Consortium (NORA) award as a Distinguished Early Career Researcher.

https://www.uio.no/dscience/english/n...