Скачать или смотреть MultiViz: Towards User-Centric Visualizations and Interpretations of Multimodal Models

MultiViz: Towards User-Centric Visualizations and Interpretations of Multimodal Models
Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-philippe Morency, Ruslan Salakhutdinov

CHI 2023: The ACM CHI Conference on Human Factors in Computing Systems
Session: Late Breaking Work (LBW) Posters B

The promise of multimodal models for real-world applications has inspired research in visualizing and understanding their internal mechanics with the end goal of empowering stakeholders to visualize model behavior, perform model debugging, and promote trust in machine learning models. However, modern multimodal models are typically black-box neural networks, which makes it challenging to understand their internal mechanics. How can we visualize the internal modeling of multimodal interactions in these models? Our paper proposes MultiViz, a method for analyzing the behavior of multimodal models via 4 stages: (1) unimodal importance, (2) cross-modal interactions, (3) multimodal representations and (4) multimodal prediction. Through studies with 21 users on 8 trained models across 6 real-world tasks, we show that the complementary stages in MultiViz together enable users to (1) simulate model predictions, (2) assign interpretable concepts to features, (3) perform error analysis on misclassifications, and (4) use insights from error analysis to debug models.

Web:: https://programs.sigchi.org/chi/2023/...

Pre-recorded presentation videos for Late-Breaking Works at CHI 2023