Скачать или смотреть CSAIL Forum, Yoon Kim, "Efficient & Expressive Architectures for Language Modeling"

This lecture was delivered by Prof. Yoon Kim on April 22, 2025, as part of the CSAIL Forum series. In this lecture, Prof. Yoon Kim discusses how Transformers have become the dominant architecture for language modeling — and for generative AI more broadly. The attention mechanism within Transformers is considered core to the architecture, enabling accurate sequence modeling at scale. However, the complexity of attention grows quadratically with input length, making it difficult to apply Transformers to model long sequences. Moreover, Transformers have theoretical limitations regarding the class of problems they can solve, preventing them from effectively modeling certain phenomena, such as state tracking. Prof. Kim describes recent work on efficient alternatives to Transformers that can overcome these limitations.