Скачать или смотреть CopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills

CopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills
Dhruva Bansal, Prerna Ravi, Matthew So, Pranay Agrawal, Ishan Chadha, Ganesh Meyyappan Murugappan, Colby T Duke

CHI '21: The 2021 ACM CHI Conference on Human Factors in Computing Systems
Session: Student Research Competition 1st Round - F

Abstract
Deaf children born to hearing parents lack continuous access to language, leading to weaker working memory compared to hearing children and deaf children born to Deaf parents. CopyCat, a game where children communicate with the computer via American Sign Language (ASL), has been shown to improve language skills and working memory. Previously, CopyCat depended on unscalable hardware such as custom gloves for sign verification. However, modern 4K cameras and pose estimators present new opportunities. Before creating the CopyCat game for deaf children using off-the-shelf hardware, we evaluate whether current ASL recognition is sufficient. Using Hidden Markov Models (HMMs), user independent word accuracies were 90.6%, 90.5%, and 90.4% for AlphaPose,Kinect, and MediaPipe respectively. Transformers performed on average 17.0% worse. Given these results, we believe our current HMM-based recognizer can be successfully adapted to verify children’s signing while playing CopyCat.

DOI:: https://doi.org/10.1145/3411763.3451523
WEB:: https://chi2021.acm.org/

Pre-recorded Presentations for the ACM CHI Virtual Conference on Human Factors in Computing Systems, May 8-13, 2021