Toward theoretical understanding of deep learning (Lecture 2) by Sanjeev Arora

DISTINGUISHED LECTURES

THREE LECTURES ON MACHINE LEARNING

SPEAKER: Sanjeev Arora (Princeton University and Institute for Advanced Study, USA)

DATE: 12 February 2019 to 13 February 2019

VENUE: Ramanujan Lecture Hall, ICTS Bangalore

Lecture 1: Mathematics of Machine Learning: An introduction
Date & Time: Tuesday, 12 February, 11:30

Abstract: Machine learning is the sub-field of computer science concerned with creating programs and machines that can improve from experience and interaction. It relies upon mathematical optimization, statistics, and algorithm design. The talk will be an introduction to machine learning for a mathematical audience. We describe the mathematical formulations of basic types of learning such as supervised, unsupervised, interactive, etc., and the philosophical and scientific issues raised by them.



Lecture 2: Toward theoretical understanding of deep learning
Date & Time: Tuesday, 12 February, 15:00

Abstract:The empirical success of deep learning drives much of the excitement about machine learning today. This success vastly outstrips our mathematical understanding. This lecture surveys progress in recent years toward developing a theory of deep learning. Works have started addressing issues such as speed of optimization, sample requirements for training, effect of architecture choices, and properties of deep generative models.


Lecture 3: Theoretical analysis of unsupervised learning
Date & Time: Wednesday, 13 February, 11:30

Abstract:Unsupervised learning refers to learning without human-labeled datapoints. This can mean many things but in this talk will primarily refer to learning representations (also called embeddings) of complicated data types such as images or text. Empirically it is possible to learn such representations which have interesting properties and also lead to better performance when combined with labeled data. This talk will survey some attempts to theoretically understand such embeddings and representations, as well as their properties. Many examples are drawn from natural language processing.

Table of Contents (powered by https://videoken.com)
0:00:00 Start
0:00:26 Toward theoretical understanding of deep learning
0:00:34 Machine learning (ML): A new kind of science
0:00:40 Recap:
0:01:31 Training via Gradient Descent ("natural algorithm")
0:02:03 Subcase: deep learning*
0:03:00 Brief history: networks of "artificial neurons"
0:04:30 Some questions
0:05:06 Part 1: Why overparameterization and/or overprovisioning?
0:05:24 Overprovisioning may help optimization (part 1): a folklore experiment
0:08:45 Overprovisioning can help (part 2): Allowing more
0:12:44 Acceleration effect of increasing depth
0:15:13 But textbooks warn us: Larger models can "Overfit"
0:19:06 Popular belief/conjecture
0:27:40 Noise stability: understanding one layer (no nonlinearity)
0:30:05 Proof sketch : Noise stability -deep net can be made low-dimensional
0:31:12 The Quantitative Bound
0:32:03 Correlation with Generalization (qualitative check)
0:32:27 Concluding thoughts on generalization
0:33:07 Part 2: Optimization in deep learning
0:36:33 Basic concepts
0:38:15 Curse of dimensionality
0:41:00 Gradient descent in unknown landscape.
0:42:14 Gradient descent in unknown landscape (contd.)
0:42:49 Evading saddle points..
0:45:28 Active area: Landscape Analysis
0:47:35 New trend: Trajectory Analysis
0:50:06 Trajectory Analysis (contd)
0:57:32 Unsupervised learning motivation: "Manifold assumption"
1:00:31 Unsupervised learning Motivation: "Manifold assumption" (contd)
1:00:45 Deep generative models
1:02:08 Generative Adversarial Nets (GANs) [Goodfellow et al. 2014]
1:05:25 What spoils a GANs trainer's day: Mode Collapse
1:08:47 Empirically detecting mode collapse (Birthday Paradox Test)
1:10:06 Estimated support size from well-known GANs
1:10:50 To wrap up....What to work on (suggestions for theorists)
1:11:43 Concluding thoughts
1:12:07 Advertisements
1:12:13 Q&A