Скачать или смотреть Upcycling LLMs into MoE with Nvidia Researcher, Ethan He

Nvidia researcher, Ethan He, joins the Oxen Herd to give a deep dive into his co-authored paper, Upcycling Large Language Models into Mixture of Experts.

--

Use Oxen AI 🐂 https://oxen.ai/

Oxen AI makes versioning your datasets as easy as versioning your code! Even is millions of unstructured images, the tool quickly handles any type of data so you can build cutting-edge AI.

--

Paper 📜 https://arxiv.org/abs/2410.07524

Links + Notes 📝 https://www.oxen.ai/blog/how-upcyclin...

Join Arxiv Dives 🤿 https://oxen.ai/community

Discord 🗿   / discord  

--

Chapters
0:00 Who is Ethan He
2:16 Ethan He Presents Upcycling LLMs
2:30 What is MoE
5:27 How Does the MoE Layer Work
8:52 How the Router Works
13:32 The auxiliary loss: Switch Transformers
14:55 Mixtral vs. Switch Transformer
18:50 The Takeaway
23:38 Plain Upcycling
28:44 Weight Scaling
32:55 Fine-Grained MoE
37:28 Fine-Grained MoE Upcycling
43:01 Experiments
43:50 The Importance of Learning Rate
45:48 Analysis of the Wave Similarity
49:43 Number of Experts
50:14 Large Scale Upcycling
55:10 Questions