Building Foundational Models from Heterogeneous and Multimodal Data Beyond Texts and Images

Foundational models (FMs), pretrained on extensive datasets using self-supervised techniques, can learn generalized patterns from vast amounts of data. This significantly reduces the need for large labeled datasets for new tasks, saving both time and resources by leveraging the broad knowledge base established during pretraining. Most research on FMs has primarily focused on unstructured unimodal data, such as text and images. In this talk, we will explore building foundational models beyond text and images, highlighting two of our projects.

The first project involves a system called OtterKnowledge (https://github.com/IBM/otter-knowledge), which fuses data from various sources, organizes the heterogeneous and multimodal data into multimodal knowledge graphs, and learns efficient representations of the data. This approach enhances pretrained unimodal models with valuable information from the graphs, resulting in improved performance on downstream tasks such as drug-target binding affinity prediction.

The second project introduces TabularFM (https://tabularfm.github.io/), a new framework that incorporates state-of-the-art methods for developing FMs specifically for tabular data. This includes variations of neural architectures such as GANs, VAEs, and Transformers. I will discuss the feasibility of building FMs for tabular data and interesting patterns that these models transfer across tabular datasets. We have curated a million tabular datasets and released cleaned versions to facilitate the development of tabular FMs. We pretrained FMs on this curated data, benchmarked various learning methods on these datasets, and released the pretrained models along with leaderboards for future comparative studies.

Biography:

Lam earned his specialist degree with honors in applied mathematics and computer science from Lomonosov Moscow State University, Russia, in 2007. In 2013, he obtained a PhD in computer science with cum laude distinction under the supervision of Professor Toon Calders at Eindhoven University of Technology (TU/e), the Netherlands. A part of his PhD research was nominated for the Best Paper Award at the SIAM SDM Conference in 2012. Since joining the IBM research lab in 2013, Lam's research has focused on representation learning, with a recent emphasis on building foundational models for science. Lam was awarded the IBM Corporate Technical Award for his research contribution to IBM AutoAI in 2021. He also holds the title of Master Inventor for his contributions to IBM patent activities. Lam has published over 26 papers in top AI conferences and is the author or co-author of 30 patents.
Personal website: https://research.ibm.com/people/thanh...