Скачать или смотреть Yapeng Tian - Audio-Visual Scene Understanding Towards Unified, Explainable, and Robust Perception

Abstract: Understanding surrounding scenes, i.e., recognizing objects, sounds, and human activities, is a fundamental capability in human intelligence. Similarly, developing computational models that can understand scenes is a central problem in AI. Humans use multiple cooperated senses with multisensory integration to understand a scene. For example, hearing helps capture the spatial location of a racing car behind us; seeing a person’s talking face can strengthen our perception on his/her speech. However, existing scene understanding algorithms are designed to solely rely on either visual or auditory modalities, and they are yet to explore whether joint audio-visual learning can facilitate understanding scenes in videos. In this talk, I will introduce a series of our works in audio-visual scene understanding for building machines with unified, explainable, and robust multisensory perception capability. At the end of the talk, I will discuss some challenges and future directions.