Скачать или смотреть Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density

The document explores how Joint Embedding Predictive Architectures (JEPAs) implicitly learn data density while aiming to produce Gaussian embeddings. It reveals that JEPAs' anti-collapse term estimates the data density. The authors demonstrate that a trained JEPA can be used to obtain sample probabilities for tasks such as data curation and outlier detection. The theoretical findings are architecture and dataset agnostic, enabling efficient computation of sample probabilities using the model's Jacobian matrix. The method, named JEPA-SCORE, is validated across synthetic, controlled, and ImageNet datasets, as well as on I-JEPA, DINOv2, and MetaCLIP models. The research shows how JEPAs perform non-parametric density estimation without input space reconstruction or a parametric model for the data density. The core idea relies on the fact that for the embedding density to be Gaussian, the model has to learn about the input data density. Ultimately, the paper reveals how to extract the implicit data density learned by JEPAs.
#JEPAs #GaussianEmbeddings #SelfSupervisedLearning #DataDensity #DensityEstimation #MetaAI #MachineLearning

paper - http://arxiv.org/pdf/2510.05949v1
subscribe - https://t.me/arxivpaper
donations:
USDT: 0xAA7B976c6A9A7ccC97A3B55B7fb353b6Cc8D1ef7
BTC: bc1q8972egrt38f5ye5klv3yye0996k2jjsz2zthpr
ETH: 0xAA7B976c6A9A7ccC97A3B55B7fb353b6Cc8D1ef7
SOL: DXnz1nd6oVm7evDJk25Z2wFSstEH8mcA1dzWDCVjUj9e
created with NotebookLM