Expectation Maximization for the Gaussian Mixture Model | Full Derivation

How to derive the EM Algorithm for the univariate Gaussian Mixture Model (GMM). Here are the handwritten notes: https://raw.githubusercontent.com/Cey...

Gaussian Mixture Models (GMMs) are extremely handy for clustering data. For example, think of clustering the grades of students after an exam into two clusters, those who passed and those who failed. For this we have to infer the parameters of the GMM (cluster-probabilities, means and standard deviations) from the latent. However, since the class node is latent we have to resort to an Expectation Maximization and the whole Maximum Likelihood Estimate will turn into an iterative procedure.

In this video we start at the derived general equations and fully derive all equations for the E-Step and the M-Step with NO EXCUSES - every derivative, manipulation and trick is presented in detail *.

The interesting observation is that although the EM implies we would need an expectation and maximization in every iteration, this is actually not the case. For the GMM, we can derive straight-forward update equations.

If something is still unclear, please write a comment :)

-------

📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): https://github.com/Ceyron/machine-lea...

📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff:   / felix-koehler   and   / felix_m_koehler  

💸 : If you want to support my work on the channel, you can become a Patreon here:   / mlsim  

-------

Timestamps:
00:00 Introduction
01:10 Clustering
01:40 Infer Parameters w\ missing data
03:05 Joint of the GMM
04:45 E-Step: Un-Normalized Responsibilities
10:29 E-Step: Normalizing the Responsibilities
11:13 M-Step: The Q-Function
15:27 M-Step: Maximization formally
16:57 M-Step: Lagrange Multiplier
20:20 M-Step: Cluster Probabilities
30:50 M-Step: Means
35:00 M-Step: Standard Deviations
39:37 Summary
42:52 Important Remark
43:37 Outro