Diffusion Models for Solving Inverse Problems (Jiaming Song, NVIDIA)

Date: Jan 31, 2023

Abstract:
Diffusion models are widely used as foundation models for generative modeling. Diffusion models can also be trained for specific inverse problems, but such models are limited to their particular use cases and are expensive to train. This talk introduces several of my recent works on using the same, generic diffusion model for solving different inverse problems. First, I will talk about Denoising Diffusion Restoration Models (DDRM) for solving noisy, linear inverse problems. Next, I will mention PhysDiff, which incorporates physical constraints into a motion diffusion model. Finally, I will introduce Pseudoinverse-guided Diffusion Models (ΠGDM). Despite using a generic diffusion model, ΠGDM achieves the performance of diffusion models specifically trained on the inverse problem. The flexibility of ΠGDM also allows it to solve a much wider set of inverse problems.

Bio:
Jiaming Song is a research scientist in the Learning and Perception Research team of NVIDIA Research. His recent research focus is on diffusion models. He created DDIM, the earliest accelerated algorithm for diffusion models that is widely used in recent generative AI systems including DALL-E 2, Imagen, Stable Diffusion, and ERNIE-ViLG 2.0. He co-authored SDEdit, the algorithm behind Stable Diffusion’s img2img method. He aims to further democratize the use of diffusion models for general applications, often in the form of inverse problems. He received his Ph.D. degree in Computer Science from Stanford University in 2021 and B.Eng. degree in Computer Science from Tsinghua University in 2016. He was a recipient of the ICLR 2022 Outstanding Paper Award.