Rayna Andreeva (07/17/2024): Metric space magnitude and generalization in neural networks

The magnitude of a metric space is a recently established invariant, providing a measure of the ‘effective size’ of a space across multiple scales. Its properties are an active area of research as it encodes many known invariants of a metric space. Surprisingly, given its richness, applications to machine learning have been scarce. In this talk, I will present an overview of the current applications, and focus on one recent development in particular: magnitude in deep learning. We study the internal representations of neural networks and propose a number of new measures based on topological complexities for determining their generalisation capabilities. These measures are computationally friendly, enabling us to propose simple yet effective algorithms for computing generalization indices. Moreover, our flexible framework can be extended to different domains, tasks, and architectures. Our experimental results demonstrate that our new complexity measures correlate highly with generalization error in industry-standards architectures such as transformers and deep graph networks. Our approach consistently outperforms existing topological bounds across a wide range of datasets, models, and optimizers, highlighting the practical relevance and effectiveness of our complexity measures.