RI: Small: Collaborative Research: Towards Modeling Source Separation from Measured Cortical Responses

该项目将使用测量大脑活动的新技术来详细了解人类听众是如何区分相互竞争、重叠的声音的，从而帮助设计能够实现同样壮举的自动系统。自然环境中充满了重叠的声音，人类和机器的成功音频处理依赖于分离感兴趣的声源的基本能力。这通常被称为“鸡尾酒会效应”，基于人们能够在嘈杂的背景音频中听到一个人在说什么。尽管在听力方面的研究历史悠久，但人类对声源分离的这种特殊能力仍然知之甚少，并且通过机器自动分离重叠声音的努力也相应地很粗糙：尽管在机器对嘈杂语音的鲁棒处理方面取得了很大进展，但分离复杂的自然声音（如重叠声音）仍然是一个挑战。现在，传感器技术的进步使我们能够对人类的这一功能进行建模，从而对大脑中的声音表征过程提供了前所未有的详细视图。该项目专门针对等待神经外科手术的患者，直接在人体皮层表面测量神经电反应（目前使用256个电极传感器阵列）。利用这种对受控混合声音的测量，该项目将努力通过从神经群体反应中重建声学刺激的近似值来开发人类皮层中的声音分离模型，并在此过程中学习神经反应与刺激的频谱测量之间的线性映射。为了尝试显著提高机器算法模仿人类源分离能力的能力，该项目还将重点放在信号处理框架上，该框架支持不同线索和策略组合的实验，以优化与神经活动记录的一致性。该工程模型基于计算听觉场景分析（CASA）框架，这是一系列处理声音混合的方法，已经显示出有竞争力的结果。

This project will use new technologies for measuring brain activity to understand in detail how human listeners are able to separate competing, overlapping voices, and thereby to help design automatic systems capable of the same feat. Natural environments are full of overlapping sounds, and successful audio processing by both humans and machines relies on a fundamental ability to separate out sound sources of interest. This is commonly referred to as the "cocktail party effect," based on the ability of people to hear what a single person is saying despite the noisy background audio from other speakers. Despite the long history of research in hearing, this exceptional human capability for sound source separation is still poorly understood, and efforts to automatically separate overlapping voices by machine are correspondingly crude: although great advances have been made in robust processing of noisy speech by machine, separation of complex natural sounds (such as overlapping voices) remains a challenge. Advances in sensor technology now enable the modeling of this function in humans, giving an unprecedented, detailed view of sound representation processing in the brain. This project works specifically with measurements of neuroelectric response made directly on the surface of the human cortex (currently with a 256-electrode sensor array) for patients awaiting neurosurgery. Using such measurements made for controlled mixtures of voices, the project will endeavor to both develop models of voice separation in the human cortex by reconstructing an approximation to the acoustic stimulus from the neural population response, and in the process learning the linear mapping between the neural response back to a spectrogram measure of the stimulus. To attempt to significantly improve the ability of machine algorithms to mimic human source separation capability, the project will also focus on a signal processing framework that supports experiments with different combinations of cues and strategies to optimize agreement with the recordings of neural activity. The engineering model is based on the Computational Auditory Scene Analysis (CASA) framework, a family of approaches that have shown competitive results for handling sound mixtures.