PonderNet in PyTorch

In this video, we implement the PonderNet that was proposed in the paper "PonderNet: Learning to Ponder". It is a network that dynamically decides on the size of its forward pass. We are going to implement it and experiment with it a little bit on the so called ParityDataset. Note that the implementation is based on the labml.ai implementaiotn (see link below). I made some modification though so I hope I did not introduce too many bugs:D

Paper: https://arxiv.org/abs/2107.05407
labml.ai annotated implementation: https://nn.labml.ai/adaptive_computat...
Code from this video: https://github.com/jankrepl/mildlyove...

00:00 Introduction
00:54 Paper explanation
05:56 Parity dataset - implementation
08:52 Parity dataset - testing it out
11:31 PonderNet - implementation
18:31 PonderNet - testing it out
19:45 Reconstruction loss
21:18 Regularization loss - implementation
23:45 Regularization loss - testing it out
24:41 Evaluation + plotting logic
27:17 Training - CLI boilerplate
28:18 Training - Setting things up
30:40 Training - train+eval loop
32:22 Experimental setup - 2 scripts
33:13 Results - experiment 1 (multiple lambdas)
38:05 Results - experiment 2 (extrapolation)
39:42 Outro


If you have any video suggestions or you just wanna chat feel free to join the discord server:   / discord  

Twitter:   / moverfitted  

Credits logo animation
Title: Conjungation · Author: Uncle Milk · Source:   / unclemilk   · License: https://creativecommons.org/licenses/... · Download (9MB): https://auboutdufil.com/?id=600