Скачать или смотреть Audio Flamingo: A Model That Understands Audio Beyond Transcriptions | ICML 2024

Augmenting large language models (LLMs) to understand audio – including non-speech sounds and non-verbal speech – is critically important for diverse real-world applications of LLMs.

In this demo video, we demonstrate Audio Flamingo, a model that can listen to audio and understand audio beyond speech transcriptions. Audio Flamingo can describe audio contents, answer questions about the audio, and chat about the audio for several rounds. Specifically, we demonstrate Audio Flamingo’s audio understanding abilities through synthetic audio that ElevenLabs generated based on OpenAI Sora’s synthetic video.

Paper: Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities. In ICML 2024. https://arxiv.org/abs/2402.01831

Authors: Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, Bryan Catanzaro.

GitHub: https://github.com/NVIDIA/audio-flamingo

Demo page: https://audioflamingo.github.io/