Why large batch sizes lead to worse generalization in Deep Learning

Intel's Research on why Small Batch sizes lead to greater generalization in Deep Learning: https://artificialintelligencemadesim...

Paper Deets
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say 32-512 data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.

https://arxiv.org/abs/1609.04836

Develop your skills-
AI Made Simple- https://artificialintelligencemadesim...
Tech Made Simple- https://codinginterviewsmadesimple.su...

Reach out to me
Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.


Small Snippets about Tech, AI and Machine Learning over here


AI Newsletter- https://artificialintelligencemadesim...


My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.su...


Check out my other articles on Medium. : https://rb.gy/zn1aiu


My YouTube: https://rb.gy/88iwdd


Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y


My Instagram: https://rb.gy/gmvuy9


My Twitter:   / machine01776819