Stochastic Depth for Neural Networks Explained

Stochastic depth is a powerful regularization method applied on residual neural networks that speed up training and enhances test performance.

In this tutorial, we'll go through the methodology and a Pytorch implementation.

Table of Content

- Introduction: 0:00
- Background and Context: 0:30
- Questions: 2:46
- Architecture Changes: 3:00
- Why use the resnet architecture for stochastic depth?: 5:07
- Difference between Drop out and Drop path: 5:23
- Speedup and performance: 6:06
- Stochastic Depth example: 6:33
- How do they manage speedup and better performance with stochastic depth?: 7:27
- Data sets: 7:44
- Main Results: 8:09
- Analytical Experiment Result: 10:10
- Code Walkthrough: 11:22
- Conclusion: 26:10

📌 Github: https://github.com/yacineMahdid/deep-...
📌 Original Code: https://github.com/shamangary/Pytorch...
📌 Paper: https://arxiv.org/pdf/1603.09382

Abstract:
"Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges.

The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time.

We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks.

It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91% on CIFAR-10)."

---
Join the Discord for general discussion:   / discord  
---

Follow Me Online Here:
GitHub: https://github.com/yacineMahdid
LinkedIn:   / yacinemahdid  

---

Have a great week! 👋