The Lottery Ticket Hypothesis and pruning in PyTorch

In this video, we are going to explain how one can do pruning in PyTorch. We will then use this knowledge to implement a paper called "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks". The paper states that feedforward neural networks have subnetworks (winning tickets) inside of them that perform as good as (or even better than) the original network. It also proposes a recipe how to find them.

Paper: https://arxiv.org/abs/1803.03635
Official code: https://github.com/facebookresearch/o...
Code from this video: https://github.com/jankrepl/mildlyove...

00:00 Intro
00:50 Paper overview: Hypothesis (diagram) [slides]
01:33 Paper overview: Hypothesis (formal) [slides]
02:15 Paper overview: Finding winning tickets [slides]
03:44 Paper overview: Our setup [slides]
05:08 Pruning 101 in PyTorch [code]
10:29 Data - MNIST [code]
12:18 Multilayer perceptron [code]
14:05 Pruning: Linear + MLP [code]
16:45 Randomly initializing: Linear + MLP [code]
18:24 Weight copying: Linear + MLP [code]
19:51 Computing statistics [code]
20:53 Training functions [code]
24:38 CLI and training preparation [code]
27:13 Train-prune loop [code]
30:04 Grid search script [code]
31:01 Results: Actual vs desired pruning [no code]
32:47 Results: Winning tickets (parallel coordinate plots) [no code]
36:02 Results: Winning tickets (standard plots) [no code]
37:12 Outro


If you have any video suggestions or you just wanna chat feel free to join the discord server:   / discord  

Twitter:   / moverfitted  

Credits logo animation
Title: Conjungation · Author: Uncle Milk · Source:   / unclemilk   · License: https://creativecommons.org/licenses/... · Download (9MB): https://auboutdufil.com/?id=600