Скачать или смотреть New way to convert any model into Mixture of Experts

Paper - https://arxiv.org/pdf/2410.07524

Discover the cutting-edge techniques for upcycling dense language models into sparse mixture-of-experts (MoE) models. In this video, we explore how to increase model capacity by leveraging efficient upcycling methods, including the novel “virtual group” initialization scheme and weight scaling approaches. We also dive into expert routing methods like softmax-then-topK, and show how higher granularity MoE architectures outperform continued dense model training. Watch as we break down the upcycling process used on Nemotron-4 15B and analyze how it compares to continuous model training. Learn how upcycling can elevate your AI projects and push the boundaries of model performance.
#moe #mixtureofexperts #mistral #mixtral #pixtral

email - [email protected]

On my way to become AI research scientist / engineer.