FractalNet Deep Neural Network Explained with Pytorch

FractalNet is an alternative to residual neural networks that was built to showcase that it wasn't the residual aspect of resnet that made it so powerful.

Indeed, the FractalNet paper was able to show that it was the training of sub-paths of different lengths that made the network much more performant than VGG.

In this tutorial, we'll go through the paper and a Pytorch implementation of FractalNet.

Table of Content
Introduction: 0:00
FractalNet Paper: 1:09
Fractal Expansion Rule: 1:44
Important Variables for Fractal Network: 3:31
Overview of Drop Path: 4:15
FractalNet Training & Datasets: 5:53
FractalNet Results: 6:28
Code Overview: 8:38
Code Introduction: 10:55
FractalNet Class: 12:42
FractalBlock Class: 18:52
FractalBlock join function: 29:56
FractalBlock drop_mask function: 30:50
ConvBlock Class: 34:20
Conclusion: 35:26

**Links*:
📌 Paper: https://arxiv.org/pdf/1605.07648
📌 Original Code: https://github.com/khanrc/pt.fractaln...
📌 Code Walkthrough: https://github.com/yacineMahdid/deep-...

**Abstract**:
"We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals.

These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers.

In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks.

Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures.

Such regularization allows extraction of high-performance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer."

----
Join the Discord for general discussion:   / discord  

----
Follow Me Online Here:

GitHub: https://github.com/yacineMahdid
LinkedIn:   / yacinemahdid  
___

Have a great week! 👋