CHS: Medium: Critical Factors for Automatic Speech Recognition in Supporting Small Group Communication Between People who are Deaf or Hard of Hearing and Hearing Colleagues

CHS：中：自动语音识别支持聋哑人或听力障碍人士与听力正常同事之间小组交流的关键因素

• 批准号: 1954284
• 负责人: Matt Huenerfauth
• 金额: $ 49.99万
• 依托单位: Rochester Institute of Tech
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1954284&HistoricalAwards=false
• 关键词: CHS; Medium; Critical; Factors; Automatic

To support the employment opportunities of people who are Deaf or Hard of Hearing (DHH), it is important that they be able to communicate effectively with colleagues who can hear. While there are regulations in many educational settings that mandate services be provided by human professionals to support the needs of people who are DHH (e.g., sign-language interpreting or real-time captioning), these services are often not provided in the workplace, where companies may only pay for limited accommodations and where meetings or impromptu chats with a co-worker often arise with little advance notice. As a consequence, DHH employees often have difficulty understanding hearing colleagues, particularly in groups, so many of them prefer to skip meetings and wait for e-mail notes afterward. Mobile "apps" using automatic speech recognition (ASR) to convert audio into text displayed on the screen of a smartphone or tablet have exciting potential for supporting live communication between DHH individuals and hearing co-workers, but even today's state of the art ASR is insufficient to this end due in part to errors that are produced in noisy, real-world settings. This project will explore the user-interface design issues that can increase the benefit to DHH individuals of the sometimes-imperfect captions produced by ASR in small-group meetings with hearing co-workers, and will determine the best ways to measure the usefulness of such technology for members of the DHH community in realistic environments. In addition to creating new scientific knowledge about how people who are DHH make use of emerging speech-recognition technologies to support communication with hearing colleagues in the workplace, project outcomes will provide guidance on the most effective design of mobile apps to enhance this communication so as to transform employment opportunities and independence for these individuals. The findings from this research will also inform other ASR dictation or communication settings, such as how designs can encourage users to speak more clearly for higher ASR accuracy.Research on ASR for live events has primarily focused on lecture transcription, but during a small group meeting participants' behavior adapts dynamically as communication unfolds. The interplay of technology and behavior during interaction often allows users to benefit from ASR even if the text output contains errors. New ASR-for-meetings apps are becoming commercially available, but prior work has revealed that naive designs lead to frustrating interactions; human-computer interaction (HCI) research with DHH users is needed to lay the groundwork for better suited technology. This project will include: surveys of people who are DHH as well as employers and hearing co-workers of DHH individuals; laboratory-based studies with individuals or small-groups to investigate the most effective user-interface designs for mobile apps for this task; participatory design and prototype usability testing of a mobile app based on these studies; and observation of DHH individuals in real-world employment settings using this prototype to determine whether the findings from lab-based studies translate to real-world use. How variations in design parameters, such as latency/speed of captioning, influence the speaking or error-correction behavior of users, and how this can be leveraged to benefit the interaction, will also be explored.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为了支持聋人或重听人（DHH）的就业机会，重要的是他们能够与听力正常的同事进行有效的沟通。 虽然在许多教育环境中有规定要求由人类专业人员提供服务，以支持DHH患者的需求（例如，这些服务通常包括手语翻译或实时字幕），但在工作场所往往不提供这些服务，公司可能只支付有限的住宿费用，而且与同事开会或即兴聊天往往没有提前通知。 因此，DHH的员工往往很难理解听力正常的同事，特别是在小组中，所以他们中的许多人宁愿跳过会议，等待会后的电子邮件说明。 移动的“应用程序”使用自动语音识别（ASR）将音频转换为显示在智能手机或平板电脑屏幕上的文本，具有支持DHH个人和听力同事之间的实时通信的令人兴奋的潜力，但即使是今天的最先进的ASR也不足以达到这一目的，部分原因是在嘈杂的现实世界环境中产生的错误。 本项目将探讨用户界面设计问题，可以增加的好处，DHH个人的有时不完美的字幕制作的ASR在小组会议与听力同事，并将确定最佳的方式来衡量这种技术的实用性DHH社区的成员在现实环境中。 除了创造关于DHH患者如何利用新兴语音识别技术支持与工作场所听力正常的同事沟通的新科学知识外，项目成果还将为移动的应用程序的最有效设计提供指导，以加强这种沟通，从而改变这些人的就业机会和独立性。 这项研究的发现也将为其他ASR听写或交流设置提供参考，例如设计如何鼓励用户更清晰地说话，以提高ASR准确性。针对现场活动的ASR研究主要集中在演讲转录上，但在小组会议期间，参与者的行为会随着交流的展开而动态调整。 交互过程中技术和行为的相互作用通常允许用户从ASR中受益，即使文本输出包含错误。 新的ASR会议应用程序正在商业化，但之前的工作表明，幼稚的设计会导致令人沮丧的交互;需要与DHH用户进行人机交互（HCI）研究，为更适合的技术奠定基础。 该项目将包括：对DHH人员以及DHH个人的雇主和听力同事进行调查;对个人或小团体进行基于实验室的研究，以调查针对这一任务的移动的应用程序的最有效用户界面设计;基于这些研究的移动的应用程序的参与式设计和原型可用性测试;并观察DHH个人在现实世界的就业环境中使用这个原型，以确定是否从实验室为基础的研究结果转化为现实世界的使用。 设计参数的变化，如字幕的延迟/速度，如何影响用户的说话或纠错行为，以及如何利用这一点来促进互动，也将被探讨。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Exploring the Design Space of Automatically Generated Emotive Captions for Deaf or Hard of Hearing Users

探索为聋哑或听力障碍用户自动生成情感字幕的设计空间

• DOI: 10.1145/3544549.3585880
• 发表时间: 2023
• 期刊: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23
• 作者: Hassan, Saad;Ding, Yao;Kerure, Agneya Abhimanyu;Miller, Christi;Burnett, John;Biondo, Emily;Gilbert, Brenden
• 通讯作者: Gilbert, Brenden

Deaf and hard-of-hearing users' preferences for hearing speakers' behavior during technology-mediated in-person and remote conversations

聋哑和听力障碍用户在技术介导的面对面和远程对话中对听力说话者行为的偏好

• DOI: 10.1145/3430263.3452430
• 发表时间: 2021
• 期刊: Proceedings of the 18th International Web for All Conference (W4A '21
• 作者: Seita, Matthew;Andrew, Sarah;Huenerfauth, Matt
• 通讯作者: Huenerfauth, Matt

Visualization of Speech Prosody and Emotion in Captions: Accessibility for Deaf and Hard-of-Hearing Users

字幕中语音韵律和情感的可视化：聋哑和听力障碍用户的辅助功能

• DOI: 10.1145/3544548.3581511
• 发表时间: 2023
• 期刊: In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
• 作者: de Lacerda Pataca, Caluã;Watkins, Matthew;Peiris, Roshan;Lee, Sooyeon;Huenerfauth, Matt
• 通讯作者: Huenerfauth, Matt

Remotely Co-Designing Features for Communication Applications using Automatic Captioning with Deaf and Hearing Pairs

使用自动字幕与失聪者和听力正常者一起远程共同设计通信应用程序的功能

• DOI: 10.1145/3491102.3501843
• 发表时间: 2022
• 期刊: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22
• 作者: Seita, Matthew;Lee, Sooyeon;Andrew, Sarah;Shinohara, Kristen;Huenerfauth, Matt
• 通讯作者: Huenerfauth, Matt

Modeling Word Importance in Conversational Transcripts: Toward improved live captioning for Deaf and hard of hearing viewers

• DOI: 10.1145/3587281.3587290
• 发表时间: 2023-04
• 期刊: Proceedings of the 20th International Web for All Conference
• 作者: Akhter Al Amin;Saad Hassan;Matt Huenerfauth;Cecilia Ovesdotter Alm