Скачать или смотреть Striveworks Journal Club: Neural Network Pruning

What Problem(s) Does Pruning Aim to Fix?


(Completely*) True Fact #1:
Neural Networks (typically) have many convolution layers (10s-100s).
Each convolution layer (typically) contains many kernels (100s-1000s).

Each kernel (typically) consists of many learned weights (100s-1000s).


(Mostly*) True Fact #2:
More kernels = more computation required to make a prediction
More computation = more power (energy) required to make a prediction
More computation = more time required to make a prediction


(Believed by most practitioners but not entirely*) True Fact #3:
More kernels (learned parameters) = more expressive network
More kernels (learned parameters) = more danger of overfitting training data


*Author genuinely believes that independent fact checkers sufficiently familiar with the topic would agree. However, owing to the time and expense involved in procuring independent reviews, the author opted to self-fact-check.



Basically, what we want is the expressiveness of a computationally intense network but without paying for it (with time and energy).