Excellence in Research: Incorporating Attention into Computational Auditory Scene Analysis Using Spectral Clustering with Focal Templates

人类表现出一种不可思议的能力，即使在干扰声音或噪音存在的情况下，也能专注于感兴趣的声音。例如，孩子的声音可能会立即被识别并做出反应--在父母保护的背景下，观鸟者可能会识别并跟踪家雀的叫声，或者商业伙伴可能会在拥挤的餐厅吃午饭时讨论最新的事件。在每个例子中，一个人将他或她的注意力集中在感兴趣的声音来源上。一旦注意力被引导，一个人就可以继续专注于这个声音，甚至到了他或她可能感觉不到(或能够被动地忽略)其他声音的程度。目前，计算听力系统无法复制这种能力。如果这种能力是可能的，计算监听系统可以为一系列应用领域做出贡献。目前的助听器技术在群体环境中严重不足，在群体环境中，噪音经常淹没听者，听者无法补偿。那些遭受听力损失的人会感到社交孤立，导致生活质量下降。如果助听器能够自动适应隔离和聚焦于显著声源，助听器的性能以及依赖助听器的人的生活将得到极大改善。有效的声学监测可以部署到与安全相关的应用程序中，例如自动检测中风患者的含糊语音，或者提醒聋人注意公共场所的重要扬声器公告。自主设备(机器人、无人机等)可以使用增强的倾听技术来指导他们的行动或行动，有可能更好地服务或保护公众。能够专注于乐团中的一种特定乐器可以改进自动音乐转录系统，并增强表演分析和培训的工具。基于感知的注意在计算听力系统中的应用是无限的，本研究将研究焦点模板在计算听觉场景分析(CASA)中的应用。CASA试图(通过计算算法和系统)重现人类仅在听觉输入下感知声学场景的能力，CASA的基本方法是根据感知的心理学原理建模的。焦点模板可以被认为是只通过符合感兴趣模式的那些听觉元素的一种动态时频过滤器。将使用频谱聚类来检测感兴趣的声音事件(如果存在于音频中)，频谱聚类已成为对集合内的“相似”元素进行分组的有用技术；这里，目标是对通过焦点模板的听觉元素进行分组。这项工作将：1)通过使用谱聚类和焦点模板在CASA中实现注意力模型，2)测量合并注意力对CASA系统在分离特定感兴趣声音方面的性能的影响，以及3)确定有效的方法来开发焦点模板，同时考虑如何扩展到“一般”收听情况。该项目包括开发“声音和音乐计算”的辅修学科，这是一个独特的教育机会，让学生在整合STEM和非STEM学科的同时为研究做出贡献。这项工作的一个成果是建立伙伴关系并加强跨学科协作者的音乐与音频计算研究团队(CROMA-TIC)。通过这些活动，该项目将：1)促进CASA的最先进水平，2)促进该领域内的合作关系，3)在林肯大学发展公认的专业知识和研究能力领域，以及4)为本科生提供通往相关职业或研究生的渠道。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Humans display an uncanny ability to focus on a sound of interest, even in the presence of interfering sounds or noise. For instance, a child’s voice may be immediately recognized and reacted-to in the context of parental protection, a bird-watcher may recognize and follow the call of a house finch, or business associates may discuss the latest events over lunch in a crowded diner. In each of these examples, a person directs his or her attention to the sound source of interest. Once attention is directed, a human can continue to focus on that sound, even to the degree that he or she may not perceive (or is able to passively ignore) other sounds. Computational listening systems, at present, are unable to replicate that ability. If such ability was possible, computational listening systems could contribute to an array of application domains. Current hearing aid technology is woefully inadequate in group settings, where noise often overwhelms the listener and the listener is unable to compensate. Those who suffer from hearing loss feel socially isolated, leading to a lower quality of life. Performance of hearing aids, and the lives of those who rely on them, would improve drastically if these devices could automatically adapt to isolate and focus on salient sound sources. Effective acoustic monitoring could be deployed to safety-related applications, such as automatically detecting slurred speech from someone suffering a stroke, or to alert the deaf to important loudspeaker announcements in a public place. Autonomous devices (robots, drones, etc.) could employ enhanced listening techniques to guide their movements or actions, potentially better serving or protecting the public. The ability to focus on a particular instrument within a musical ensemble could lead to improved automatic music transcription systems and enhanced tools for performance analysis and training. The applications of computational listening systems exhibiting perceptually-based attention are boundless.This research will investigate the use of focal templates to incorporate attention into computational auditory scene analysis (CASA). CASA attempts to reproduce (via computational algorithms and systems) the ability of humans to perceive an acoustic scene given only auditory input, and the underlying methods of CASA are modeled upon psychological principles of perception. A focal template may be considered a type of dynamic time-frequency filter that passes only those auditory elements conforming to a pattern of interest. Sound events of interest (if present in the audio) will be detected using spectral clustering, which has emerged as a useful technique for grouping "like" elements within a set; here, the goal is to group auditory elements that pass through the focal template. This work will: 1) implement a model of attention into CASA through use of spectral clustering with focal templates, 2) measure the impact of incorporating attention on the performance of CASA systems in isolating particular sounds of interest, and 3) determine effective methods to develop focal templates while considering how to scale to the "general" listening case. This project includes development of an academic minor in "Sound and Music Computing", a unique educational opportunity to prepare students to contribute to the research while integrating across STEM and non-STEM disciplines. An outcome of this work is to build partnerships and strengthen the Computational Research on Music & Audio Team of Interdisciplinary Collaborators (CRoMA-TIC). Through these activities, the project will: 1) advance the state of the art in CASA, 2) foster collaborative relationships within this field, 3) develop a well-recognized area of expertise and research capacity at Lincoln University, and 4) develop a pipeline for undergraduate students leading to related careers or graduate study.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.