ICML 2021 | Modern Hopfield Networks - Dr Sepp Hochreiter

We propose a new paradigm for deep learning by equipping each layer of a deep-learning architecture with modern Hopfield networks. The new paradigm comprises functionalities like pooling, memory, and attention for each layer. Recently, we saw a renaissance of Hopfield Networks with tremendously increased storage capacity and convergence in one update step while ensuring global convergence a local energy minimum. Surprisingly, the transformer attention mechanism is equal to modern Hopfield Networks. In layers of deep learning architectures, they allow the storage of, and the access to, raw input data, intermediate results, reference data, or learned prototypes. These Hopfield layers enable new ways of deep learning and provide pooling, memory, nearest-neighbor, set association, and attention mechanisms. We apply deep networks with Hopfield layers to various domains, where they improve the state of the art on different tasks and for numerous benchmarks.


Dr Hochreiter is a pioneer in the field of Artificial Intelligence (AI). He was the first to identify the key obstacle to Deep Learning and then discovered a general approach to address this challenge. He thus became the founding father of modern Deep Learning and AI. He is also a professor at Johannes Kepler University Linz.


Subscribe to our newsletter and stay in the know:
https://www.iarai.ac.at/events/

___________________________________________________________________
IARAI | Institute of Advanced Research in Artificial Intelligence
www.iarai.ac.at