CHS: Medium: Collaborative Research: Wearable Sound Sensing and Feedback Techniques for Persons who are Deaf or Hard of Hearing

CHS：媒介：合作研究：适用于聋哑人的可穿戴声音传感和反馈技术

• 批准号: 1763199
• 负责人: Leah Findlater
• 金额: $ 90万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1763199&HistoricalAwards=false
• 关键词: CHS; Medium; Collaborative; Research; Wearable

The goal of this research is to advance wearable sound sensing and feedback technology for users who are deaf or hard of hearing, along with appropriate techniques for human-computer interaction. A wearable device with multiple microphones and audio processing algorithms to automatically sense, localize, and identify sounds will be developed. Means to discreetly provide this information to the user via emerging wearable technologies such as head-mounted displays and smartwatches will also be implemented. Evaluations will include lab and field studies, everyday tasks such as noticing sounds and participating in oral conversations, and objective and subjective measures. Project outcomes will have broad impact by enabling new sound awareness options for users who are deaf or hard of hearing, thereby augmenting the wearer's existing strategies with additional, unobtrusive information. The new assistive technologies will have the potential to improve the lives of a large portion of the population, in particular the growing number of older adults with hearing loss in the United States.To these ends, major subgoals will include: understanding user needs for wearable sound sensing and feedback, including prioritizing the importance of different sounds across a variety of contexts and based on an individual user's level of hearing loss; developing and evaluating a lightweight wearable sound sensing platform and accompanying algorithms, including both new sound scene analysis algorithms for a microphone array conformal on, or in proximity to, a complex-shaped baffle (the wearer's head) and that take into account how sound scatters off the wearer's body, along with adaptive state-of-the-art sound classification and speech recognition approaches to work with this processed audio; and developing and evaluating visual and haptic or vibrational feedback of the sensed sound via wearable prototypes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

本研究的目标是推进可穿戴声音传感和反馈技术的用户谁是聋人或重听，沿着适当的人机交互技术。 将开发具有多个麦克风和音频处理算法的可穿戴设备，以自动感知，定位和识别声音。 通过新兴的可穿戴技术（如头戴式显示器和智能手表）谨慎地向用户提供这些信息的方法也将被实施。 评估将包括实验室和实地研究，日常任务，如注意声音和参与口头对话，以及客观和主观的措施。项目成果将产生广泛的影响，为失聪或听力困难的用户提供新的声音感知选项，从而用额外的、不显眼的信息增强佩戴者的现有策略。新的辅助技术将有可能改善大部分人口的生活，特别是美国越来越多的老年听力损失患者。为此，主要的子目标将包括：了解用户对可穿戴声音感测和反馈的需求，包括在各种环境中并基于个体用户的听力损失水平对不同声音的重要性进行优先级排序;开发和评估轻型可穿戴声音传感平台和相关算法，包括用于麦克风阵列的新声音场景分析算法，该麦克风阵列共形于复杂形状的挡板上或附近（佩戴者的头部）并且考虑声音如何从佩戴者的身体散射，沿着自适应最先进的声音分类和语音识别方法来处理该处理的音频;通过可穿戴原型开发和评估感知声音的视觉和触觉或振动反馈。该奖项反映了NSF的法定使命，并被认为是值得的。通过使用基金会的知识价值和更广泛的影响审查标准进行评估来提供支持。

项目成果

Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study

探索实时字幕的增强现实方法：一项初步的民族志研究

• DOI: 10.1145/3197391.3205404
• 发表时间: 2018
• 期刊: Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems
• 作者: Jain, Dhruv;Chinh, Bonnie;Findlater, Leah;Kushalnagar, Raja;Froehlich, Jon
• 通讯作者: Froehlich, Jon

Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing

• DOI: 10.1145/3463501
• 发表时间: 2021-06
• 期刊: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
• 作者: Steven M. Goodman;Ping Liu;Emma J. McDonnell;Jon E. Froehlich;Steven M. Goodman;Ping Liu;D. Jain;Emma J. McDonnell;Jon E. Froehlich

SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users

• DOI: 10.1145/3373625.3416991
• 发表时间: 2020-10
• 期刊: Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility
• 作者: D. Jain;Hung Ngo;Pratyush Patel;Steven M. Goodman;Leah Findlater;Jon E. Froehlich

HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing Users

• DOI: 10.1145/3313831.3376758
• 发表时间: 2020-04
• 期刊: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
• 作者: D. Jain;Kelly Avery Mack;Akli Amrous;Matt Wright;Steven M. Goodman;Leah Findlater;Jon E. Froehlich

“Easier or Harder, Depending on Who the Hearing Person Is”: Codesigning Videoconferencing Tools for Small Groups with Mixed Hearing Status

– 更容易或更难，取决于听力正常的人是谁 – 为具有混合听力状态的小群体共同设计视频会议工具

• DOI: 10.1145/3544548.3580809
• 发表时间: 2023
• 期刊: Proceedings of ACM CHI 2023
• 作者: McDonnell, Emma J;Moon, Soo Hyun;Jiang, Lucy;Goodman, Steven M.;Kushalnagar, Raja;Froehlich, Jon E.;Findlater, Leah
• 通讯作者: Findlater, Leah