Andreas Mueller - MotherNet: A Foundational Hypernetwork for Tabular Classification

Title: MotherNet: A Foundational Hypernetwork for Tabular Classification

Abstract:

Recently, Prior Fitted Networks, and in particular TabPFN showed how to train transformer architectures on collections of synthetic datasets to enable in-context learning of classification and other tasks on new tabular datasets. Building on this work, we introduce MotherNet, a hypernetwork architecture that can produce a small neural network via in-context learning. This approach is somewhat surprising, since it provides an entirely new way of generating weights for a neural network, without backpropagation. We find that the models created using in-context learning by MotherNet outperform neural networks trained with Adam using optimizer hyper-parameters. We also find that MotherNet performs competitive with XGBoost, at faster training times and without any hyper-parameter tuning.

Speaker: Andreas Müller - https://amueller.github.io/

Paper: https://arxiv.org/abs/2312.08598