RM+ML: 26. Gradient Descent for Linear Regression

The lecture notes for the course can be found at https://rolandspeicher.com/wp-content...
neural network, gradient descent, linear regression, random feature model, learning algorithm

0:00 Learning and gradient descent
6:35 Feature learning
8:58 Gradient descent for linear regression
36:29 Gradient descent algorithm
42:02 Example
56:22 Ridged regression

The goal of this lecture series is to cover mathematical interesting aspects of neural networks, in particular, those related to random matrices. In this 26th lecture we start to look on the gradient descent learning algorithm. In order to get a feeling for it we look on the simplest model, namely linear regression in the under-determined regime. For this we have an explicit formula for the solution and we can check whether and how gradient descent converges to this.