How to Train Neural Networks Fast and Efficiently | Tutorial

0:00 Multi-GPU Training
2:15 Cyclic Learning Rate Schedules
3:07 Mixup: Beyond Empirical Risk Minimization
3:44 Label Smoothing
4:28 Deep Double Descent
5:55 Transfer Learning
6:18 Mixed Precision Training (Theory)
10:00 Mixed Precision Training (Tutorial)

@CodeEmporium (Ajay's channel)
   / @codeemporium  

Code for the video
https://gist.github.com/ajhalthor/140...

Code behind the DCGAN with Apex
https://github.com/NVIDIA/apex/blob/m...

Mixed Precision Training
https://arxiv.org/abs/1710.03740

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
https://arxiv.org/abs/1706.02677

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
https://arxiv.org/abs/1807.11205

Don't Decay the Learning Rate, Increase the Batch Size
https://openreview.net/pdf?id=B1Yy1BxCZ

On the Variance of the Adaptive Learning Rate and Beyond
https://arxiv.org/abs/1908.03265

Cyclical Learning Rates for Training Neural Networks
https://arxiv.org/abs/1506.01186

The 1cycle policy
https://sgugger.github.io/the-1cycle-...

Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates
https://arxiv.org/pdf/1708.07120.pdf

Bag of Tricks for Image Classification with Convolutional Neural Networks
https://arxiv.org/pdf/1812.01187.pdf

mixup: Beyond Empirical Risk Minimization
https://arxiv.org/abs/1710.09412

Deep Double Descent
https://openai.com/blog/deep-double-d...

Deep Double Descent: Where Bigger Models and More Data Hurt
https://arxiv.org/abs/1912.02292

Reconciling modern machine learning practice and the bias-variance trade-of
https://arxiv.org/abs/1812.11118