Why do we divide by n-1 to estimate the variance? A visual tour through Bessel correction

Correction: At 30:42 I write "X = Y". They're not equal, what I meant to say is "X and Y are identically distributed".

The variance is a measure of how spread out a distribution is. In order to estimate the variance, one takes a sample of n points from the distribution, and calculate the average square deviation from the mean.
However, this doesn't give a good estimate of the variance of the distribution. The best estimate, however, is obtained when dividing by n-1 instead of n.
WHY!?!?!?!?!?!?!?
In this video, we dig deeper into why the variance calculation should be divided by n-1 instead of by n. For this, we use an alternate definition of the variance, which doesn't use the mean in its calculation.

[0:00] Introduction and Bessel's Correction
- Introducing Bessel's Correction and why we divide by \( n-1 \) instead of \( n \) to estimate variance.

[0:12] Introduction to Variance Calculation
- Explaining the premise of calculating variance and introducing the concept of estimating variance using a sample instead of the entire population.

[1:01] Definition of Variance
- Defining variance as a measure of how much values deviate from the mean and outlining the basic steps of variance calculation.

[1:52] Introduction to Bessel's Correction
- Discussing why we divide by \( n-1 \) when calculating variance and introducing Bessel's Correction.

[2:35] Challenges of Bessel's Correction
- Sharing personal challenges in understanding the rationale behind Bessel's Correction and discussing my research process on the topic.

[3:20] Alternative Definition of Variance
- Presenting an alternative definition of variance to aid in understanding Bessel's Correction and expressing curiosity about its presence in the literature.

[4:45] Quick Recap of Mean and Variance
- Briefly revisiting the concepts of mean and variance, demonstrating how they are calculated with examples, and explaining how variance reflects different distributions.

[7:05] Sample Mean and Variance Estimation
- Explaining the challenges of estimating the mean and variance of a distribution using a sample and discussing why sample variance is not a good estimate.

[8:49] Bessel's Correction and Why \( n-1 \) is Used
- Explaining how Bessel's Correction provides a better estimate of variance and why we divide by \( n-1 \) instead of \( n \). Emphasizing the importance of making a correct variance estimate.

[10:51] Why Better Estimation Matters?
- Discussing why the original estimate is poor and why making a better estimate is crucial. Explaining the significance of sample mean as a good estimate.

[13:02] Issues with Variance Estimation
- Illustrating the problems with variance estimation and demonstrating with examples why using the correct mean is essential for accurate estimates. Explaining the accuracy of estimates made using \( n-1 \).

[15:04] Introduction to Correcting the Estimate
- Discussing the underestimated variance and the need for correction in estimation.

[15:57] Adjusting the Variance Formula
- Explaining the adjustment in the variance formula by changing the denominator from \( n \) to \( n - 1 \).

[16:22] Calculation Illustration
- Demonstrating the calculation process of variance with the adjusted formula using examples.

[16:57] Better Estimate with Bessel's Correction
- Discussing how the corrected estimate provides a more accurate variance estimation.

[18:24] New Method for Variance Calculation
- Introducing a new method for calculating variance without explicitly calculating the mean.

[20:06] Understanding the Relation between Variance and Variance
- Explaining the relationship between variance and variance, and how they are related mathematically.

[21:52] Demonstrating a Bad Calculation
- Illustrating a flawed method for calculating variance and explaining the need for correction.

[23:37] The Role of Bessel's Correction
- Explaining why removing unnecessary zeros in variance calculation leads to better estimates, equivalent to Bessel's Correction.

[25:08] Summary of Estimation Methods
- Summarizing the difference between the flawed and corrected estimation methods for variance.

[26:02] Importance of Bessel's Correction
- Emphasizing the significance of Bessel's Correction for accurate variance estimation, especially with smaller sample sizes.

[30:19] Mathematical Proof of Variance Relationship
- Providing two proofs of the relationship between variance and variance, highlighting their equivalence.

[35:24] Acknowledgments and Conclusion

Thanks @mkan543 for the summary!