Lecture 4 - Automatic Differentiation

Lecture 4 of the online course Deep Learning Systems: Algorithms and Implementation.

This lecture introduces automatic differentiation. We will go through numerical differentiation and gradient checking, forward mode automatic differentiation, and reverse mode automatic differentiation.

Sign up for the course for free at http://dlsyscourse.org.

Contents
00:00 - Introduction
00:48 - How does differentiation fit into machine learning
05:11 - Numerical differentiation
11:36 - Numerical gradient checking
14:06 - Symbolic differentiation
18:27 - Computational graph
22:47 - Forward mode automatic differentiation (AD)
28:25 - Limitations of forward mode AD
29:45 - Reverse mode automatic differentiation (AD)
36:34 - Derivation for the multiple pathway case
40:14 - Reverse AD algorithm
43:40 - Reverse mode AD by extending the computational graph
51:50 - Reverse mode AD vs Backprop
57:40 - Reverse mode AD on Tensors
01:00:12 - Reverse mode AD on data structures