Modified Newton method | Exact Line Search | Theory and Python Code | Optimization Algorithms #4

☕️ Buy me a coffee: https://paypal.me/donationlink240
🙏🏻 Support me on Patreon:   / ahmadbazzi  

In this one, I will show you what the modified newton algorithm is and how to use it with the exact line search method. We will approach both methods from intuitive and animated perspectives. The difference between Damped and its modified newton method is that the Hessian may run into singularities at some iterations, and so we apply diagonal loading, or Tikhonov regularization at each iteration. As a reminder, Damped newton, just like newton’s method, makes a local quadratic approximation of the function based on information from the current point, and then jumps to the minimum of that approximation. Just imagine fitting a little quadratic surface in higher dimensions to your surface at the current point, and then going to the minimum of the approximation to find the next point. Finding the direction towards the minimum of the quadratic approximation is what you are doing. As a matter of fact, this animation shows you why in certain cases, Newton's method can converge to a saddle or a maximum. If the eigenvalues of the Hessian are non positive - in those cases the local quadratic approximation is an upside down paraboloid.


⏲Outline⏲
00:00 Introduction
00:55 Modified Newton Method
03:41 Exact line search
04:55 Python Implementation
18:46 Animation Module
34:14 Animating Iterations
37:02 Outro


📚Related Courses:
📚 Convex Optimization Extended Course    • Convex Optimization  
📚 Python Programming Extended Course    • Python Programming  
📚 Convex Optimization Applications Extended Course    • The Transshipment Problem in Decision...  
📚 Linear Algebra Extended Course    • Linear Algebra  
📚 Python projects course    • Python  





🔴 Subscribe for more videos on CUDA programming
👍 Smash that like button, in case you find this tutorial useful.
👁‍🗨 Speak up and comment, I am all ears.


💰 If you are able to, donate to help the channel
Patreon -   / ahmadbazzi  

#python #optimization #algorithm