AIDRC Seminar Series - Dr. Mohammad Amin Sadeghi

Dr. Mohammad Amin Sadeghi, Qatar Computing Research Institute

Date: March 27, 2023

Talk Topic: WhiteGrad: A fast neural network optimizer

Abstract: Second-order optimizers converge with fewer steps than first-order optimizers. So why are second-order optimizers not being used for deep learning? Because each step of their learning is computationally expensive. In this talk, I will present a few techniques to efficiently use second-order optimization ideas for deep learning. This optimizer, WhiteGrad, converges faster than SGD and Adam. It uses a form of whitening that involves matrix multiplication. We propose a decomposition technique that makes whitening fast. In my talk, I will present the intuition and algorithm behind this optimizer. I will also show some promising results on learning convolutional networks and vision transformers. I will finally discuss a range of new ideas that become possible with this optimizer.

Bio: Amin Sadeghi completed his PhD in Machine Learning at UIUC in 2015. Currently he is a senior scientist at Qatar Computer Research Institute. He has a broad experience in Machine Learning, Computer Vision and NLP. He has work experience at Google and Amazon. He also has a track record of creating and exiting from multiple successful AI-based startups. He is a recipient of the Bronze Medal at the IOI. He is also the only holder of CVPR best student paper award in the middle east.