CAREER: Inclusive, Private Mobile Input and Interaction Using Lip Reading

移动设备上的语音和耳语输入可以为包括视力低下和失明的人在内的广泛用户提供快速、无缝的免提输入和交互。但在许多情况下，由于环境噪音或出于隐私和安全方面的考虑，说话和窃窃私语是不可行的，甚至只是为了不打扰他人。一种通过视觉解释嘴唇动作来理解语音的系统，即基于图像的唇读或无声语音，可以缓解其中的许多挑战。然而，静音语音识别系统通常比常见的语音识别模型更慢、更容易出错，而且它们可能需要在现实世界中不切实际的硬件。因此，到目前为止，这种方法还没有被作为移动设备上重要的替代交互模式进行研究，也不知道如何最好地设计无声语音的用户界面，或者可以增强其可用性的反馈类型。失明的人也没有很好地研究无声语言。这项研究将开发一种高效的实时嘴唇阅读器，它使用移动设备的前置摄像头来捕捉嘴唇的运动，并将其翻译成文本。特别关注的是直观用户界面的设计，该界面提供了一系列视觉、听觉和触觉反馈，以促进无错误的文本输入。向更广泛的用户提供移动设备接入将产生更广泛的影响，例如语言障碍者或哑巴用户。最终，项目成果可以在虚拟现实、汽车用户界面和许多其他系统中得到利用，以提高其可用性、保密性、安全性和可访问性。当用户通过使用深度3D卷积神经网络(3D-CNN)、递归网络和连接主义时间分类损失来默默说话时，实时唇形阅读器将对来自移动摄像机的实时视频馈送进行切片和重叠，以一次识别一个音素。它将增加一个精炼通道，使用深度去噪自动编码器(DDA)和自定义语言模型检测、自动更正字符和单词级别的错误并提供反馈。将开发一系列听觉和触觉反馈，以促进无错误输入，并为视力低下和失明的人提供不间断的相机观看。该项目还将通过利用语音、无声语音和触摸交互来开发多模式纠错方法。最后，它将构建一个无声语音识别API，用于设计和开发无障碍移动输入和交互技术。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Speech and whisper input on mobile devices can offer fast and seamless hands-free input and interaction to a wide range of users, including people with low vision and blindness. But there are many scenarios where speech and whisper are not viable due to ambient noise or because of privacy and security concerns, or even simply not to disturb other people. A system that understands speech by visually interpreting lip movements, known as image-based lip reading or silent speech, can mitigate many of these challenges. However, silent speech recognition systems are typically slower and more error prone than common speech recognition models, and they may require hardware that is impractical in real-world scenarios. Hence to date this approach has not been investigated as a serious alternative mode of interaction on mobile devices, and it is unknown how best to design the user interface for silent speech or the types of feedback that can enhance its usability. Silent speech has also not been well studied with people without sight. This research will develop an efficient real-time lip reader that uses the front camera of a mobile device to capture the motion of the lips and interprets that into text. A particular focus is on the design of an intuitive user interface that provides a range of visual, auditory, and tactile feedback to facilitate error free text entry. Even broader impacts will derive from providing access to mobile devices to a wider range of users, such as persons with speech disorders or who are mute. Ultimately, project outcomes could be exploited in virtual reality, automotive user interfaces, and many other systems to increase their usability, privacy, security and accessibility. The real-time lip reader will slice and overlap live video feeds from a mobile camera to recognize one phoneme at a time as the user silently speaks by using a deep 3D convolutional neural network (3D-CNN), a recurrent network, and the connectionist temporal classification loss. It will be augmented with a refiner channel that will detect, auto-correct and provide feedback on both character and word-level errors using deep denoising autoencoder (DDA) and custom language models. A range of auditory and tactile feedback will be developed to facilitate error free input and uninterrupted camera view for people with low vision and blindness. The project will also develop multi-modal error correction approaches by exploiting speech, silent speech, and touch interactions. Finally, it will build a silent speech recognition API for the design and development of accessible mobile input and interaction techniques.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.