EchoWhisper: Exploring an Acoustic-based Silent Speech Interface for Smartphone Users

EchoWhisper: Exploring an Acoustic-based Silent Speech Interface for Smartphone Users
Yang Gao, Yincheng Jin, Jiyang Li, Seokmin Choi, Zhanpeng Jin

UbiComp '20: The ACM International Joint Conference on Pervasive and Ubiquitous Computing 2020
Session: Speech interaction + Fabrication

Abstract
With the rapid growth of artificial intelligence and mobile computing, intelligent speech interface has recently become one of the prevalent trends and has already presented huge potentials to the public. To address the privacy leakage issue during the speech interaction or accommodate some special demands, silent speech interfaces have been proposed to enable people's communication without vocalizing their sound (e.g., lip reading, tongue tracking). However, most existing silent speech mechanisms require either background illuminations or additional wearable devices. In this study, we propose the EchoWhisper as a novel user-friendly, smartphone-based silent speech interface. The proposed technique takes advantage of the micro-Doppler effect of the acoustic wave resulting from mouth and tongue movements and assesses the acoustic features of beamformed reflected echoes captured by the dual microphones in the smartphone. Using human subjects who perform a daily conversation task with over 45 different words, our system can achieve a WER (word error rate) of 8.33%, which shows the effectiveness of inferring silent speech content. Moreover, EchoWhisper has also demonstrated its reliability and robustness to a variety of configuration settings and environmental factors, such as smartphone orientations and distances, ambient noises, body motions, and so on.

DOI:: https://doi.org/10.1145/3411830
WEB:: https://ubicomp.org/ubicomp2020/

Remote Presentations for ACM International Joint Conference on Pervasive and Ubiquitous Computing 2020 (UbiComp '20)