CHS: Medium: Critical Factors for Automatic Speech Recognition in Supporting Small Group Communication Between People who are Deaf or Hard of Hearing and Hearing Colleagues

为了支持聋人或重听人(DHH)的就业机会，重要的是他们能够有效地与听力正常的同事沟通。虽然在许多教育环境中，有法规要求由人类专业人员提供服务，以支持DHH患者的需求(例如，手语翻译或实时字幕)，但这些服务通常不在工作场所提供，因为在工作场所，公司可能只支付有限的住宿费用，而且经常在几乎没有事先通知的情况下与同事举行会议或即兴聊天。因此，DHH的员工经常难以听懂同事的声音，特别是在小组中，所以他们中的许多人更喜欢跳过会议，等待之后的电子邮件记录。使用自动语音识别(ASR)将音频转换为显示在智能手机或平板电脑屏幕上的文本的移动应用程序在支持DHH个人和听力同事之间的实时交流方面具有令人兴奋的潜力，但即使是今天的最先进的ASR也不足以达到这一目的，部分原因是在嘈杂的真实世界环境中产生的错误。这个项目将探索用户界面设计问题，以增加由ASR在与听力同事的小组会议中制作的有时不完美的字幕对DHH个人的好处，并将确定在现实环境中衡量这种技术对DHH社区成员的有用性的最佳方式。除了创造关于DHH患者如何利用新兴的语音识别技术来支持在工作场所与听力正常的同事进行沟通的新科学知识外，项目成果还将就如何最有效地设计移动应用程序来加强这种沟通提供指导，以改变这些人的就业机会和独立性。这项研究的发现还将为其他ASR听写或交流设置提供参考，例如设计如何鼓励用户更清晰地发言，以获得更高的ASR准确性。对于现场活动的ASR的研究主要集中在演讲转录上，但在小组会议中，参与者的行为会随着交流的展开而动态适应。交互过程中技术和行为的相互作用通常允许用户从ASR中受益，即使文本输出包含错误。新的ASR-for-Meetings应用程序正在商业化，但先前的研究表明，简单的设计会导致令人沮丧的交互；需要对DHH用户进行人机交互(HCI)研究，为更适合的技术奠定基础。该项目将包括：对DHH个人以及DHH个人的雇主和听力同事的调查；对个人或小组进行基于实验室的研究，以调查针对这项任务的移动应用程序的最有效用户界面设计；基于这些研究对移动应用程序的参与式设计和原型可用性测试；以及使用该原型观察DHH个人在真实世界就业环境中的情况，以确定基于实验室研究的结果是否转化为真实世界的使用。还将探讨设计参数的变化，如字幕的延迟/速度，如何影响用户的说话或纠错行为，以及如何利用这一点来促进互动。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

To support the employment opportunities of people who are Deaf or Hard of Hearing (DHH), it is important that they be able to communicate effectively with colleagues who can hear. While there are regulations in many educational settings that mandate services be provided by human professionals to support the needs of people who are DHH (e.g., sign-language interpreting or real-time captioning), these services are often not provided in the workplace, where companies may only pay for limited accommodations and where meetings or impromptu chats with a co-worker often arise with little advance notice. As a consequence, DHH employees often have difficulty understanding hearing colleagues, particularly in groups, so many of them prefer to skip meetings and wait for e-mail notes afterward. Mobile "apps" using automatic speech recognition (ASR) to convert audio into text displayed on the screen of a smartphone or tablet have exciting potential for supporting live communication between DHH individuals and hearing co-workers, but even today's state of the art ASR is insufficient to this end due in part to errors that are produced in noisy, real-world settings. This project will explore the user-interface design issues that can increase the benefit to DHH individuals of the sometimes-imperfect captions produced by ASR in small-group meetings with hearing co-workers, and will determine the best ways to measure the usefulness of such technology for members of the DHH community in realistic environments. In addition to creating new scientific knowledge about how people who are DHH make use of emerging speech-recognition technologies to support communication with hearing colleagues in the workplace, project outcomes will provide guidance on the most effective design of mobile apps to enhance this communication so as to transform employment opportunities and independence for these individuals. The findings from this research will also inform other ASR dictation or communication settings, such as how designs can encourage users to speak more clearly for higher ASR accuracy.Research on ASR for live events has primarily focused on lecture transcription, but during a small group meeting participants' behavior adapts dynamically as communication unfolds. The interplay of technology and behavior during interaction often allows users to benefit from ASR even if the text output contains errors. New ASR-for-meetings apps are becoming commercially available, but prior work has revealed that naive designs lead to frustrating interactions; human-computer interaction (HCI) research with DHH users is needed to lay the groundwork for better suited technology. This project will include: surveys of people who are DHH as well as employers and hearing co-workers of DHH individuals; laboratory-based studies with individuals or small-groups to investigate the most effective user-interface designs for mobile apps for this task; participatory design and prototype usability testing of a mobile app based on these studies; and observation of DHH individuals in real-world employment settings using this prototype to determine whether the findings from lab-based studies translate to real-world use. How variations in design parameters, such as latency/speed of captioning, influence the speaking or error-correction behavior of users, and how this can be leveraged to benefit the interaction, will also be explored.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.