Mean Field Approach for Variational Inference | Intuition & General Derivation

Variational Inference tries to fit a surrogate posterior to mimic the true posterior. But how should we choose the surrogate posterior? Here are the notes: https://raw.githubusercontent.com/Cey...

In an earlier video, we saw that we can solve the optimization problem in Variational Inference that consisted of minimizing the KL by maximizing the ELBO. That allowed us to avoid the usage of the posterior, which we do not know (otherwise we would not be doing Variational Inference in the first place). But the question remained: What is q? Or in other terms: What kind of surrogate posterior should I choose.

Spoiler: This video will not answer this question ;) Here, we will just derive a general result for the idea of subdividing the surrogate posterior into smaller independent distributions. But that is an important finding, which then let us find functional forms of the surrogate naturally based on our problem.

-------

📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): https://github.com/Ceyron/machine-lea...

📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff:   / felix-koehler   and   / felix_m_koehler  

💸 : If you want to support my work on the channel, you can become a Patreon here:   / mlsim  

-------

Timestamps:
00:00 Introduction
00:45 Recap: Variational Inference
01:07 Definition: Mean Field Approach
02:19 But, what is Q?
02:55 Example for 3d latent vector
03:20 ELBO Maximization for the example
03:51 Recap: Evidence Lower Bound
05:12 Factorization plugged into ELBO
06:14 Simplifying the ELBO for q_0
14:24 Special Expectation Notation
15:37 Simplifying the ELBO for q_0 (cont.)
17:33 Simplified ELBO in optimization
18:58 Maximizing the Functional
23:07 Generalization for arbitrary subdivisions
24:31 Summary
25:02 Outro