Скачать или смотреть Understanding 2017 Google MOE

Google 2017 MOE paper: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
https://arxiv.org/abs/1701.06538

This paper proposed solutions for three problems in scaling MOE training.
1. Shrinking batch size
2. Low Compute-Communication Ratio
3. Load-Imbalance of expert networks

Timeline:
Agenda: 00:00:35
What is MOE: 00:01:10
MOE setup in the paper: 00:02:47
Distributed training - Data Parallel: 00:03:38
Distributed training - Compute-communication Ratio: 00:04:48
Distributed training - Model parallel: 00:06:11
Conditional computation: 00:06:58
Three MOE bottlenecks: 00:07:49
Shrinking batch size problem: 00:08:08
Low Compute-Communication Ratio: 00:11:10
Load imbalance of experts: 00:12:33
Experiment results: 00:13:38