XLSTM - Extended LSTMs with sLSTM and mLSTM (paper explained)

LSTMs or Long Short Term Memory Networks have been around for a long time. They have been applied for quite a few sequence-related tasks such as text generation and translation. Their drawback has been that they couldn't be parallelized and so they couldn't make use of the power of GPUs. These limitations of LSTMs have paved the way for the emergence of transformers that leveraged powerful GPUs for massive parallelization of training and inference.
What if we now go back to LSTMs, make them parallel, and see how they compare with transformers? This is the exact question answered by this current paper XLSTM which stands for extended Long short-term memory.
This video explain the two main building blocks of the XLSTM namely sLSTM and mLSTM. We delve deep into the equations and in the end see the results.

⌚️ ⌚️ ⌚️ TIMESTAMPS ⌚️ ⌚️ ⌚️
0:00 - Intro and overview of XLSTM
0:20 - Problems with LSTMs
1:29 - Recurrent Neural Networks (RNNs)
2:08 - LSTMs overview
3:39 - Drawbacks of LSTMs
4:49 - Sigmoid vs Exponential Function
6:01 - sLSTM block
7:24 - Normalizer in sLSTM
8:01 - Stabilizer in sLSTM
8:32 - mLSTM block
9:56 - Detailed block of sLSTM
10:38 - Detailed block of mLSTM
11:31 - XLSTM
11:50 - Advantages of XLSTM
12:00 - Evaluation

RELATED LINKS
XLSTM paper - https://arxiv.org/abs/2405.04517
LSTM paper - https://www.bioinf.jku.at/publication...

OUR KEY LINKS
YouTube:    / @aibites  
Twitter:   / ai_bites​  
Patreon:   / ai_bites​  
Github: https://github.com/ai-bites​

WHO AM I?
I am a Machine Learning researcher/practitioner who has seen the grind of academia and start-ups. I started my career as a software engineer 15 years ago. Because of my love for Mathematics (coupled with a glimmer of luck), I graduated with a Master's in Computer Vision and Robotics in 2016 when the now happening AI revolution started. Life has changed for the better ever since.

#machinelearning #deeplearning #aibites