NCS-FR: Engineering Brain Circuits for Complex Scene Analysis

日常社交场合，如拥挤的聚会、餐厅、教室或开放式工作场所，都会有多个演讲者和听众，以及背景噪音的嗡嗡声。在这些复杂的声音环境中，具有典型听力的人类能够识别和倾听个别声源，例如单个扬声器在说什么，而忽略其他声源，例如其他人的电话或背景噪音，比如汽车在街上行驶。这是一个被称为复杂场景分析(CSA)的一般性问题的例子，对于来自神经科学、计算机科学、语音识别和工程学等不同领域的科学家来说，即使经过50多年的研究，完全理解具有典型听力的人类如何解决这个问题仍然是难以实现的。正因为如此，CSA仍然是许多人的问题，比如听力受损的人，助听器等医疗设备的问题，以及自动语音识别系统等技术的问题。该项目研究了典型听力中复杂场景分析的神经基础，并在这些发现的基础上，开发了一种大脑启发的CSA算法。该项目最终将通过各种应用程序改善生活质量，例如提高助听器和语音识别技术的有效性。解决这一问题需要跨学科的努力，作为研究的一部分，开发了一个教育平台，以培训学生整合来自不同学科的知识，使他们能够更好地解决具有挑战性的重要社会问题。该项目整合了三个跨学科的研究思路，以开发大脑启发算法。第一个线程使用人类进行CSA的大脑成像，使用集成的可穿戴设备测量大脑信号(功能性近红外光谱和脑电)，以及机器学习方法来解码复杂视听场景中受试者所在的位置。第二条线索研究注意状态下CSA的大脑皮层回路，这被认为可以提高CSA的性能。这条线索整合了电生理学、光遗传学、行为和小鼠的计算建模，这是一个拥有成熟、强大的实验工具的模型系统，用于解开大脑皮层电路。第三个线程为可穿戴设备设计了注意力引导算法，该算法在复杂场景中选择性地处理受关注的信号源，整合了从受试者大脑信号解码的受关注位置(线程1)，以及注意力状态下的大脑皮层电路模型(线程2)。这个线程优化了算法，为CSA生成快速、紧凑、节能和最先进的算法，并在人类身上评估其性能。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Everyday social situations, like a crowded party, a restaurant, a classroom, or an open-plan workplaces, involve multiple speakers and listeners and the hum of background noise. In these complex sound environments, humans with typical hearing are able to identify and listening to individual sound sources, for example what a single speaker is saying, while ignoring the other sound sources, for example someone else's phone call or background noise, like cars driving down the street. This is an example of a general problem called complex scene analysis (CSA), and a full understanding of how humans with typical hearing solve this problem has remained elusive to scientists from a diverse range of fields - neuroscience, computer science, speech recognition and engineering - even after more than 50 years of research. Because of this, CSA remains a problem for many humans, like those with hearing impairment, for medical devices, like hearing aids, and for technology, for example automatic speech recognition systems. This project investigates the neural basis of complex scene analysis in typical hearing, and, based on these discoveries, develops a brain inspired algorithm for CSA. This project will ultimately improve quality of life through a variety of applications, for example for improving the effectiveness of hearing aids and speech recognition technologies. Solving this problem requires an interdisciplinary effort, and as part of the research, an educational platform is developed to train students to integrate knowledge from a variety of disciplines that makes them better able to address challenging and important societal problems. This project integrates three interdisciplinary research threads to develop the brain-inspired algorithm. The first thread uses brain imaging in humans performing CSA with an integrated wearable device that measures brain signals (functional near-infrared spectroscopy and electroencephalography), and machine learning methods to decode where a subject is attending in a complex audiovisual scene. The second thread investigates cortical circuits for CSA in attentive states, which are thought to enhance CSA performance. This thread integrates electrophysiology, optogenetics, behavior and computational modeling in mice, a model system with well-established, powerful experimental tools for unraveling cortical circuits. The third thread designs an attention steered algorithm for the wearable device that selectively processes an attended source in a complex scene, integrating the attended location decoded from a subject’s brain signals (thread 1), and a model of cortical circuits in attentive states (thread 2). This thread optimizes the algorithm to generate a fast, compact, energy efficient, and state of the art algorithm for CSA and evaluate its performance in humans.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.