Скачать или смотреть CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question

CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question
Danna Gurari, Kristen Grauman

CHI'17: ACM CHI Conference on Human Factors in Computing Systems
Session: Collaborative Crowdwork

Abstract:
Visual question answering systems empower users to ask any question about any image and receive a valid answer. However, existing systems do not yet account for the fact that a visual question can lead to a single answer or multiple different answers. While a crowd often agrees, disagreements do arise for many reasons including that visual questions are ambiguous, subjective, or difficult. We propose a model, CrowdVerge, for automatically predicting from a visual question whether a crowd would agree on one answer. We then propose how to exploit these predictions in a novel application to efficiently collect all valid answers to visual questions. Specifically, we solicit fewer human responses when answer agreement is expected and more human responses otherwise. Experiments on 121,811 visual questions asked by sighted and blind people show that, compared to existing crowdsourcing systems, our system captures the same answer diversity with typically 14-23% less crowd involvement.

DOI: http://dx.doi.org/10.1145/3025453.302...
WEB: http://chi2017.acm.org/