Скачать или смотреть Adaptive Learning Rate Algorithms - Yoni Iny @ Upsolver (Eng)

Slides: http://bit.ly/1LAw8iY

The standard backpropagation algorithm shows a very slow rate of convergence and a high dependency on the value of the learning rate parameter .
The shape of the multi dimensional error function for the majority of the applications usually presents irregularities which are difficult to handle using gradient descent technique with a fixed step size .
Another serious problem is the existence of regions with local minima.
If the search procedure enters such a region, the minimization will be directed to that minimum and stop when it reaches it, while it would be desired that it continues towards a global minimum, or at least a sufficiently low one .
It was discovered that the appropriate manipulation of the learning rate parameter during the training process, can lead to very good results and hence a large number of different methods for its adaptation have been proposed.
During the meetup, Yoni will explain stochastic gradient descent and various adaptive learning methods including: Fixed, fixed with annealing, Momentum, AdaGrad, AdaDelta, CG and VSGD.