CHS: Medium: Collaborative Research: Wearable Sound Sensing and Feedback Techniques for Persons who are Deaf or Hard of Hearing

CHS：媒介：合作研究：适用于聋哑人的可穿戴声音传感和反馈技术

• 批准号: 1763219
• 负责人: Raja Kushalnagar
• 金额: $ 10万
• 依托单位: Gallaudet University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1763219&HistoricalAwards=false
• 关键词: CHS; Medium; Collaborative; Research; Wearable

The goal of this research is to advance wearable sound sensing and feedback technology for users who are deaf or hard of hearing, along with appropriate techniques for human-computer interaction. A wearable device with multiple microphones and audio processing algorithms to automatically sense, localize, and identify sounds will be developed. Means to discreetly provide this information to the user via emerging wearable technologies such as head-mounted displays and smartwatches will also be implemented. Evaluations will include lab and field studies, everyday tasks such as noticing sounds and participating in oral conversations, and objective and subjective measures. Project outcomes will have broad impact by enabling new sound awareness options for users who are deaf or hard of hearing, thereby augmenting the wearer's existing strategies with additional, unobtrusive information. The new assistive technologies will have the potential to improve the lives of a large portion of the population, in particular the growing number of older adults with hearing loss in the United States.To these ends, major subgoals will include: understanding user needs for wearable sound sensing and feedback, including prioritizing the importance of different sounds across a variety of contexts and based on an individual user's level of hearing loss; developing and evaluating a lightweight wearable sound sensing platform and accompanying algorithms, including both new sound scene analysis algorithms for a microphone array conformal on, or in proximity to, a complex-shaped baffle (the wearer's head) and that take into account how sound scatters off the wearer's body, along with adaptive state-of-the-art sound classification and speech recognition approaches to work with this processed audio; and developing and evaluating visual and haptic or vibrational feedback of the sensed sound via wearable prototypes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

本研究的目标是推进可穿戴声音传感和反馈技术的用户谁是聋人或重听，沿着适当的人机交互技术。 将开发具有多个麦克风和音频处理算法的可穿戴设备，以自动感知，定位和识别声音。 通过新兴的可穿戴技术（如头戴式显示器和智能手表）谨慎地向用户提供这些信息的方法也将被实施。 评估将包括实验室和实地研究，日常任务，如注意声音和参与口头对话，以及客观和主观的措施。项目成果将产生广泛的影响，为失聪或听力困难的用户提供新的声音感知选项，从而用额外的、不显眼的信息增强佩戴者的现有策略。新的辅助技术将有可能改善大部分人口的生活，特别是美国越来越多的听力损失老年人的生活。为此，主要子目标将包括：了解用户对可穿戴声音感测和反馈的需求，包括在各种环境中并基于个体用户的听力损失水平对不同声音的重要性进行优先级排序;开发和评估轻型可穿戴声音传感平台和相关算法，包括用于麦克风阵列的新声音场景分析算法，该麦克风阵列共形于复杂形状的挡板上或附近（佩戴者的头部）并且考虑声音如何从佩戴者的身体散射，沿着自适应最先进的声音分类和语音识别方法来处理该处理的音频;通过可穿戴原型开发和评估感知声音的视觉和触觉或振动反馈。该奖项反映了NSF的法定使命，并被认为是值得的。通过使用基金会的知识价值和更广泛的影响审查标准进行评估来提供支持。

项目成果

mmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals

• DOI: 10.1145/3381010
• 发表时间: 2020-03-01
• 期刊: PROCEEDINGS OF THE ACM ON INTERACTIVE MOBILE WEARABLE AND UBIQUITOUS TECHNOLOGIES-IMWUT
• 作者: Santhalingam, Panneer Selvam;Hosain, Al Amin;Kushalnagar, Raja
• 通讯作者: Kushalnagar, Raja

Teleconference Accessibility and Guidelines for Deaf and Hard of Hearing Users

聋哑和听力障碍用户的电话会议辅助功能和指南

• DOI: 10.1145/3373625.3417299
• 发表时间: 2020
• 期刊: ASSETS '20: The 22nd International ACM SIGACCESS Conference on Computers and Accessibility
• 作者: Kushalnagar, Raja S.;Vogler, Christian
• 通讯作者: Vogler, Christian

Mixed Reality Speaker Identification as an Accessibility Tool for Deaf and Hard of Hearing Users

混合现实说话人识别作为聋哑和听力障碍用户的辅助工具

• DOI: 10.1145/3359996.3364720
• 发表时间: 2019
• 期刊: VRST '19: 25th ACM Symposium on Virtual Reality Software and Technology
• 作者: Glasser, Abraham;Mason Riley, Edward;Weeks, Kaitlyn;Kushalnagar, Raja
• 通讯作者: Kushalnagar, Raja

Effect of caption width on the TV user experience by deaf and hard of hearing viewers

字幕宽度对聋哑和听力障碍观众电视用户体验的影响

• DOI: 10.1145/3430263.3452435
• 发表时间: 2021
• 期刊: W4A '21: Proceedings of the 18th International Web for All Conference
• 作者: Glasser, Abraham;Garcia, Joseline;Hwang, Chang;Vogler, Christian;Kushalnagar, Raja
• 通讯作者: Kushalnagar, Raja

Live Captions in Virtual Reality (VR)

虚拟现实 (VR) 中的实时字幕

• 发表时间: 2023
• 期刊: Journal on Technology & Persons with Disabilities
• 作者: Pranav Pidathala, Dawson Franz