Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency

Liyue Shen
Assistant Professor of Electrical and Computer Engineering
University of Michigan, College of Engineering

Abstract: Diffusion models have recently emerged as powerful generative priors for solving inverse problems. However, training diffusion models in the pixel space are data-intensive and computationally demanding, which restricts their applicability in domains such as medical imaging. Latent diffusion models, which operate in a lower-dimensional space, offer a potential solution to these challenges. Though, their direct application to solving inverse problems remains an unsolved technical challenge due to the nonlinearity of the encoder and decoder. In this talk, I will introduce our recently proposed algorithm named ReSample, which can solve general inverse problems with latent diffusion models. This algorithm incorporates data consistency by solving an optimization problem during the reverse sampling process, a concept that we term as hard data consistency. I will show the considerable efficiency and flexibility offered by this approach with high-quality image reconstruction in both linear and non-linear inverse problems, as well as an easy generalization to different domains by simply fine-tuning the pre-trained latent diffusion model with a minimal amount of data samples. Finally, I will provide further discussion about our interesting ongoing works with continuous efforts to develop computationally efficient generative diffusion models for high-resolution and high-dimensional data including 3D and dynamic medical images. This research opens the door to leverage generative diffusion models in tackling complex real-world data for addressing various crucial problems in many scientific disciplines.

Bio: Liyue Shen is an assistant professor in the EECS department at the University of Michigan. Prior to that, she received her B.E. degree in Electronic Engineering from Tsinghua University in 2016 and obtained her Ph.D. degree from the Department of Electrical Engineering, Stanford University in 2022. She also spent one year as a postdoctoral research fellow at the Department of Biomedical Informatics, Harvard Medical School. Her research interest is in Biomedical AI, which lies in the interdisciplinary areas of machine learning, computer vision, signal and image processing, biomedical imaging, medical image analysis, and data science. She recently focuses on the generative diffusion models, implicit neural representation learning, and multimodal foundation models. She is the recipient of the Stanford Bio-X Bowes Graduate Student Fellowship (2019-2022) and was selected as the Rising Star in EECS by MIT and the Rising Star in Data Science by the University of Chicago in 2021. Website: https://liyueshen.engin.umich.edu/