Modified Newton method | Backtracking Armijo | Theory and Python Code | Optimization Techniques #5

In this one, I will show you what the modified newton algorithm is and how to use it with the backtracking search by Armijo rule. We will approach both methods from intuitive and animated perspectives. The difference between Damped and its modified newton method is that the Hessian may run into singularities at some iterations, and so we apply diagonal loading, or Tikhonov regularization at each iteration. As a reminder, Damped newton, just like newton’s method, makes a local quadratic approximation of the function based on information from the current point, and then jumps to the minimum of that approximation. Just imagine fitting a little quadratic surface in higher dimensions to your surface at the current point, and then going to the minimum of the approximation to find the next point. Finding the direction towards the minimum of the quadratic approximation is what you are doing. As a matter of fact, this animation shows you why in certain cases, Newton's method can converge to a saddle or a maximum. If the eigenvalues of the Hessian are non positive - in those cases the local quadratic approximation is an upside down paraboloid. Next, we talk about the line search we are going to use in this tutorial, which is the Armijo backtracking method. This is achieved by the Armijo condition, which sufficiently decreases our function ! Of course, looking at the Armijo condition equation as is might not reveal any insights, but geometrically looks beautiful, let me show you how.


⏲Outline⏲
00:00 Introduction
00:57 Modified Newton Method
03:44 Backtracking by Armijo
06:41 Python Implementation
24:41 Animation Module
40:12 Animating Iterations
43:32 Outro


📚Related Courses:
📚 Convex Optimization Extended Course    • Convex Optimization  
📚 Python Programming Extended Course    • Python Programming  
📚 Convex Optimization Applications Extended Course    • The Transshipment Problem in Decision...  
📚 Linear Algebra Extended Course    • Linear Algebra  
📚 Python projects course    • Python  





🔴 Subscribe for more videos on CUDA programming
👍 Smash that like button, in case you find this tutorial useful.
👁‍🗨 Speak up and comment, I am all ears.


💰 If you are able to, donate to help the channel
Patreon -   / ahmadbazzi  
BTC wallet - 3KnwXkMZB4v5iMWjhf1c9B9LMTKeUQ5viP
ETH wallet - 0x44F561fE3830321833dFC93FC1B29916005bC23f
DOGE wallet - DEvDM7Pgxg6PaStTtueuzNSfpw556vXSEW
API3 wallet - 0xe447602C3073b77550C65D2372386809ff19515b
DOT wallet - 15tz1fgucf8t1hAdKpUEVy8oSR8QorAkTkDhojhACD3A4ECr
ARPA wallet - 0xf54bEe325b3653Bd5931cEc13b23D58d1dee8Dfd
QNT wallet - 0xDbfe00E5cddb72158069DFaDE8Efe2A4d737BBAC
AAVE wallet - 0xD9Db74ac7feFA7c83479E585d999E356487667c1
AGLD wallet - 0xF203e39cB3EadDfaF3d11fba6dD8597B4B3972Be
AERGO wallet - 0xd847D9a2EE4a25Ff7836eDCd77E5005cc2E76060
AST wallet - 0x296321FB0FE1A4dE9F33c5e4734a13fe437E55Cd
DASH wallet - XtzYFYDPCNfGzJ1z3kG3eudCwdP9fj3fyE

This lecture contains many optimization techniques.

#python #optimization #algorithm