Скачать или смотреть Machine Learning NeEDS Mathematical Optimization with Prof Ce Zhang

Abstract: Today’s machine learning models are trained dominantly in a centralized environment such as data centers. Such a strong dependency on closely coupled interconnections is causing problems not only with the cost of infrastructure, contributing to the staggering cost of model training (not uncommon to be in the million-dollar regime!) but also with the transparency and openness of today’s machine learning ecosystem. While a decentralized paradigm has been successfully applied to various other areas such as SETI@HOME and Folding@HOME, their adoption to machine learning comes with great challenges — today’s optimization algorithms that are designed for fast networks are inherently communication heavy. To bring AI into a decentralized future, we need to fundamentally rethink the training algorithm, system design, and hardware accelerations, all together.

In this talk, I will provide a personal reflection on our efforts in decentralized learning over the last decade, and put them into the context of many fascinating work that has been conducted by the community during the same period of time. I will also discuss my personal view on the future of decentralized learning and a “cry for help” with many challenges that we are still struggling with.