Скачать или смотреть A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Eva...

A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Eva...
Glen Berman, Nitesh Goyal, Michael Madaio

CHI 2024: The ACM CHI Conference on Human Factors in Computing Systems
Session: Ethics of Digital Technologies B

Responsible design of AI systems is a shared goal across HCI and AI communities. Responsible AI (RAI) tools have been developed to support practitioners to identify, assess, and mitigate ethical issues during AI development. These tools take many forms (e.g., design playbooks, software toolkits, documentation protocols). However, research suggests that use of RAI tools is shaped by organizational contexts, raising questions about how effective such tools are in practice. To better understand how RAI tools are—and might be—evaluated, we conducted a qualitative analysis of 37 publications that discuss evaluations of RAI tools. We find that most evaluations focus on usability, while questions of tools’ effectiveness in changing AI development are sidelined. While usability evaluations are an important approach to evaluate RAI tools, we draw on evaluation approaches from other fields to highlight developer- and community-level steps to support evaluations of RAI tools’ effectiveness in shaping AI development practices and outcomes.



Web:: https://programs.sigchi.org/chi/2024/...

Pre-recorded video presentations for Papers at CHI 2024