Скачать или смотреть Gumbel Softmax Quantization: Differentiable Discrete Sampling

Code: https://github.com/priyammaz/PyTorch-...

Gumbel Softmax Quantization is a differentiable alternative to the standard Vector Quantization we saw before. The reason this is important is we get to avoid that whole pairwise distance and Argmin operation we do for VQVAEs, and get clean gradients to update our codebook without any codebook/commitment loss!

Today we start with the Gumbel-Max trick, derive its relation to Softmax, and then extend to the Gumbel Softmax from there! The derivation we do is pretty close to the example given by the Princeton CS group (https://lips.cs.princeton.edu/the-gum...) but I fill in some of the mathematical details they skipped!

I hope you already have seen the following pre-reqs:
VAE:    • Variational AutoEncoders (VAE) Implementation  
VQVAE:    • Vector Quantized Variational AutoEncoder (...  

Timestamps:
00:00:00 - Introduction
00:00:30 - Recap VQVAE
00:02:32 - Selecting Codes is a Sampling Problem
00:04:20 - What is the Gumbel Distribution?
00:09:08 - The Gumbel-Max Trick vs Multinomial Sampling
00:13:08 - Prove Gumbel-Max is Equivalent to Multinomial Sampling
00:39:24 - How to Sample from Gumbel Distribution?
00:43:24 - Simulation to Show Equivalence
00:44:50 - Moving to Gumbel Softmax
00:48:00 - Implementing Gumbel Softmax
00:50:00 - Effect of the Temperature Parameter Tau
00:52:36 - Implement One-Hot-Encoded Outputs (Hard Outputs)
00:58:03 - Implement Gumbel Softmax Quantizer
01:06:00 - Training Model with Annealed Tau
01:10:51 - Importance in Wav2Vec2

Socials!
X   / data_adventurer  
Instagram   / nixielights  
Linkedin   / priyammaz  
Discord   / discord  
🚀 Github: https://github.com/priyammaz
🌐 Website: https://www.priyammazumdar.com/