Excellence in Research: Incorporating Attention into Computational Auditory Scene Analysis Using Spectral Clustering with Focal Templates

卓越研究：使用带有焦点模板的谱聚类将注意力纳入计算听觉场景分析

• 批准号: 2100874
• 负责人: David Heise
• 金额: $ 49.9万
• 依托单位: Lincoln University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2100874&HistoricalAwards=false
• 关键词: Excellence; Research; Incorporating; Attention; into

Humans display an uncanny ability to focus on a sound of interest, even in the presence of interfering sounds or noise. For instance, a child’s voice may be immediately recognized and reacted-to in the context of parental protection, a bird-watcher may recognize and follow the call of a house finch, or business associates may discuss the latest events over lunch in a crowded diner. In each of these examples, a person directs his or her attention to the sound source of interest. Once attention is directed, a human can continue to focus on that sound, even to the degree that he or she may not perceive (or is able to passively ignore) other sounds. Computational listening systems, at present, are unable to replicate that ability. If such ability was possible, computational listening systems could contribute to an array of application domains. Current hearing aid technology is woefully inadequate in group settings, where noise often overwhelms the listener and the listener is unable to compensate. Those who suffer from hearing loss feel socially isolated, leading to a lower quality of life. Performance of hearing aids, and the lives of those who rely on them, would improve drastically if these devices could automatically adapt to isolate and focus on salient sound sources. Effective acoustic monitoring could be deployed to safety-related applications, such as automatically detecting slurred speech from someone suffering a stroke, or to alert the deaf to important loudspeaker announcements in a public place. Autonomous devices (robots, drones, etc.) could employ enhanced listening techniques to guide their movements or actions, potentially better serving or protecting the public. The ability to focus on a particular instrument within a musical ensemble could lead to improved automatic music transcription systems and enhanced tools for performance analysis and training. The applications of computational listening systems exhibiting perceptually-based attention are boundless.This research will investigate the use of focal templates to incorporate attention into computational auditory scene analysis (CASA). CASA attempts to reproduce (via computational algorithms and systems) the ability of humans to perceive an acoustic scene given only auditory input, and the underlying methods of CASA are modeled upon psychological principles of perception. A focal template may be considered a type of dynamic time-frequency filter that passes only those auditory elements conforming to a pattern of interest. Sound events of interest (if present in the audio) will be detected using spectral clustering, which has emerged as a useful technique for grouping "like" elements within a set; here, the goal is to group auditory elements that pass through the focal template. This work will: 1) implement a model of attention into CASA through use of spectral clustering with focal templates, 2) measure the impact of incorporating attention on the performance of CASA systems in isolating particular sounds of interest, and 3) determine effective methods to develop focal templates while considering how to scale to the "general" listening case. This project includes development of an academic minor in "Sound and Music Computing", a unique educational opportunity to prepare students to contribute to the research while integrating across STEM and non-STEM disciplines. An outcome of this work is to build partnerships and strengthen the Computational Research on Music & Audio Team of Interdisciplinary Collaborators (CRoMA-TIC). Through these activities, the project will: 1) advance the state of the art in CASA, 2) foster collaborative relationships within this field, 3) develop a well-recognized area of expertise and research capacity at Lincoln University, and 4) develop a pipeline for undergraduate students leading to related careers or graduate study.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人类表现出一种不可思议的能力，即使在干扰声音或噪音的存在下，也能专注于感兴趣的声音。 例如，在父母保护的背景下，孩子的声音可以立即被识别并做出反应，观鸟者可以识别并跟随家雀的叫声，或者商业伙伴可以在拥挤的晚餐中讨论最新的事件。 在这些例子中的每一个中，一个人将他或她的注意力引导到感兴趣的声源。 一旦注意力被引导，人类可以继续专注于那个声音，甚至到他或她可能无法感知（或能够被动忽略）其他声音的程度。 目前，计算听力系统无法复制这种能力。 如果这种能力是可能的，计算听系统可以有助于一系列的应用领域。 目前的助听器技术在群体环境中是远远不够的，在群体环境中，噪音经常困扰听者，听者无法补偿。 听力损失患者会感到社会孤立，导致生活质量下降。 如果助听器能够自动适应隔离并专注于突出的声源，助听器的性能以及依赖它们的人的生活将大大改善。 有效的声学监测可以部署到与安全相关的应用中，例如自动检测中风患者的口齿不清，或者提醒聋人注意公共场所的重要扬声器广播。 自主设备（机器人、无人机等）可以使用增强的监听技术来指导他们的动作或行动，从而更好地服务或保护公众。 在音乐合奏中专注于特定乐器的能力可能会导致改进的自动音乐转录系统和增强的性能分析和培训工具。 基于感知的注意的计算听觉系统的应用是无限的，本研究将探讨焦点模板的使用，将注意纳入计算听觉场景分析（CASA）。 CASA试图（通过计算算法和系统）再现人类感知听觉场景的能力，CASA的基本方法是基于感知的心理学原理。 焦点模板可以被认为是一种动态时频滤波器，其仅通过符合感兴趣模式的那些听觉元素。 感兴趣的声音事件（如果存在于音频中）将使用谱聚类来检测，谱聚类已经成为一种有用的技术，用于对集合内的“相似”元素进行分组;在这里，目标是对通过焦点模板的听觉元素进行分组。 这项工作将：1）通过使用具有焦点模板的谱聚类将注意力模型实现到CASA中，2）测量在分离感兴趣的特定声音时将注意力并入对CASA系统的性能的影响，以及3）确定开发焦点模板的有效方法，同时考虑如何缩放到“一般”收听情况。 该项目包括“声音和音乐计算”的学术未成年人的发展，这是一个独特的教育机会，可以帮助学生为研究做出贡献，同时整合STEM和非STEM学科。 这项工作的成果是建立伙伴关系，加强跨学科合作者音乐音频团队的计算研究（CRoMA-TIC）。 通过这些活动，该项目将：1）推进CASA的最新技术，2）促进该领域内的合作关系，3）在林肯大学发展公认的专业知识和研究能力领域，和4）为本科生发展相关职业或研究生学习的管道。该奖项反映了NSF的法定使命，并被认为是值得支持的，使用基金会的知识价值和更广泛的影响审查标准进行评估。

项目成果

Visually Exploring Multi-Purpose Audio Data

可视化探索多用途音频数据

• DOI: 10.1109/mmsp53017.2021.9733552
• 发表时间: 2021
• 期刊: 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP
• 作者: Heise, David;Bear, Helen L.
• 通讯作者: Bear, Helen L.