Excellence in Research: Incorporating Attention into Computational Auditory Scene Analysis Using Spectral Clustering with Focal Templates

人类表现出一种不可思议的能力，即使在干扰声音或噪音存在的情况下，也能专注于感兴趣的声音。例如，在父母的保护下，孩子的声音可能会被立即识别并做出反应，观鸟者可能会识别并跟踪家雀的叫声，或者商业伙伴可能会在拥挤的餐馆里讨论午餐时发生的最新事件。在这些例子中，一个人将他或她的注意力引导到感兴趣的声源上。一旦注意力被引导，一个人就可以继续把注意力集中在那个声音上，甚至到了他或她可能察觉不到（或能够被动地忽略）其他声音的程度。目前，计算机侦听系统还无法复制这种能力。如果这种能力是可能的，计算侦听系统可以为一系列应用领域做出贡献。目前的助听器技术在群体环境中严重不足，在群体环境中，噪音经常压倒听者，听者无法补偿。那些患有听力损失的人感到社会孤立，导致生活质量下降。如果这些设备能够自动适应隔离和聚焦突出的声源，助听器的性能和依赖它们的人的生活将会得到极大的改善。有效的声学监测可以部署到与安全相关的应用中，比如自动检测中风患者的口齿不清，或者提醒聋人注意公共场所的重要广播。自主设备（机器人、无人机等）可以采用增强的监听技术来指导它们的移动或行动，从而更好地服务或保护公众。在音乐合奏中专注于特定乐器的能力可能会导致改进的自动音乐转录系统和增强的表演分析和训练工具。计算聆听系统表现出基于感知的注意力的应用是无限的。本研究将探讨焦点模板的使用，将注意力纳入计算听觉场景分析（CASA）。CASA试图（通过计算算法和系统）重现人类仅在听觉输入的情况下感知声学场景的能力，CASA的基本方法以感知的心理学原理为基础。焦点模板可以被认为是一种动态时频滤波器，它只通过那些符合感兴趣模式的听觉元素。感兴趣的声音事件（如果存在于音频中）将使用频谱聚类来检测，这已经成为一种有用的技术，用于将“相似”元素分组在一个集合中；在这里，目标是对通过焦点模板的听觉元素进行分组。这项工作将：1)通过使用焦点模板的频谱聚类将注意力模型应用到CASA中；2)测量将注意力纳入CASA系统在隔离特定感兴趣的声音方面对性能的影响；3)确定开发焦点模板的有效方法，同时考虑如何扩展到“一般”听力案例。该项目包括开设辅修课程“声音和音乐计算”，这是一个独特的教育机会，可以帮助学生在整合STEM和非STEM学科的同时为研究做出贡献。这项工作的一个成果是建立伙伴关系并加强跨学科合作者的音乐和音频计算研究团队（CRoMA-TIC）。通过这些活动，该项目将：1)提高CASA的技术水平，2)促进该领域内的合作关系，3)在林肯大学发展公认的专业知识和研究能力，4)为本科生提供通往相关职业或研究生学习的管道。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Humans display an uncanny ability to focus on a sound of interest, even in the presence of interfering sounds or noise. For instance, a child’s voice may be immediately recognized and reacted-to in the context of parental protection, a bird-watcher may recognize and follow the call of a house finch, or business associates may discuss the latest events over lunch in a crowded diner. In each of these examples, a person directs his or her attention to the sound source of interest. Once attention is directed, a human can continue to focus on that sound, even to the degree that he or she may not perceive (or is able to passively ignore) other sounds. Computational listening systems, at present, are unable to replicate that ability. If such ability was possible, computational listening systems could contribute to an array of application domains. Current hearing aid technology is woefully inadequate in group settings, where noise often overwhelms the listener and the listener is unable to compensate. Those who suffer from hearing loss feel socially isolated, leading to a lower quality of life. Performance of hearing aids, and the lives of those who rely on them, would improve drastically if these devices could automatically adapt to isolate and focus on salient sound sources. Effective acoustic monitoring could be deployed to safety-related applications, such as automatically detecting slurred speech from someone suffering a stroke, or to alert the deaf to important loudspeaker announcements in a public place. Autonomous devices (robots, drones, etc.) could employ enhanced listening techniques to guide their movements or actions, potentially better serving or protecting the public. The ability to focus on a particular instrument within a musical ensemble could lead to improved automatic music transcription systems and enhanced tools for performance analysis and training. The applications of computational listening systems exhibiting perceptually-based attention are boundless.This research will investigate the use of focal templates to incorporate attention into computational auditory scene analysis (CASA). CASA attempts to reproduce (via computational algorithms and systems) the ability of humans to perceive an acoustic scene given only auditory input, and the underlying methods of CASA are modeled upon psychological principles of perception. A focal template may be considered a type of dynamic time-frequency filter that passes only those auditory elements conforming to a pattern of interest. Sound events of interest (if present in the audio) will be detected using spectral clustering, which has emerged as a useful technique for grouping "like" elements within a set; here, the goal is to group auditory elements that pass through the focal template. This work will: 1) implement a model of attention into CASA through use of spectral clustering with focal templates, 2) measure the impact of incorporating attention on the performance of CASA systems in isolating particular sounds of interest, and 3) determine effective methods to develop focal templates while considering how to scale to the "general" listening case. This project includes development of an academic minor in "Sound and Music Computing", a unique educational opportunity to prepare students to contribute to the research while integrating across STEM and non-STEM disciplines. An outcome of this work is to build partnerships and strengthen the Computational Research on Music & Audio Team of Interdisciplinary Collaborators (CRoMA-TIC). Through these activities, the project will: 1) advance the state of the art in CASA, 2) foster collaborative relationships within this field, 3) develop a well-recognized area of expertise and research capacity at Lincoln University, and 4) develop a pipeline for undergraduate students leading to related careers or graduate study.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.