[서울대 AI 여름학교] Google Research 김승연 - Why Distillation Helps: A Statistical Perspective

강연 제목: Why Distillation Helps: A Statistical Perspective
강연자: Google Research 김승연 연구원

서울대 AI 연구원은 2021년 8월 제2회 'AI 여름학교'를 개최하여 국내외 저명한 AI 연구자들의 강연을 모든 사람들에게 온라인으로 제공하였습니다. 3일간 1만명이 등교했던 여름학교 강의를 만나보시기 바랍니다.

Abstract: Knowledge distillation is a technique for improving the performance of a simple "student" model by replacing its one-hot training labels with a distribution over labels obtained from a complex "teacher" model. While this simple approach has proven widely effective, a basic question remains unresolved: why does distillation help? This talk presents a statistical perspective on distillation which addresses this question, and provides a novel connection to extreme multiclass retrieval techniques. Our core observation is that the teacher seeks to estimate the underlying (Bayes) class-probability function. Building on this, we establish a fundamental bias-variance tradeoff in the student's objective: this quantifies how approximate knowledge of these class-probabilities can significantly aid learning. Finally, we show how distillation complements existing negative mining techniques for extreme multiclass retrieval, and propose a unified objective which combines these ideas.

제2회 서울대 AI 여름학교 https://aiis.snu.ac.kr/aisummerschool...
제1회 서울대 AI 여름학교 https://aiis.snu.ac.kr/aisummerschool...