RI: Medium: Neuromorphic and Data-Driven Speech Segregation

RI：中：神经形态和数据驱动的语音分离

• 批准号: 1764010
• 负责人: Shihab Shamma
• 金额: $ 85.17万
• 依托单位: University of Maryland, College Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1764010&HistoricalAwards=false
• 关键词: RI; Medium; Neuromorphic; Data; Driven

This project investigates how neural representations of speech and music in the cortex can be adapted and applied to overcome the challenge of robust perception in extremely noisy and cluttered environments, mimicking processing and capabilities of the brain. More specifically, the project will formulate algorithms inspired by the architecture of the brain to segregate and track targeted speakers or sound sources, test their performance, and relate them to state-of-the-art approaches that utilize deep artificial neural networks to accomplish these tasks. Human psychoacoustic and physiological experiments with these algorithms will be conducted to test the validity of these ideas for mimicking human abilities. This effort will spur the development of new neuromorphic computational tools modeled after the brain and its cognitive functions. In turn, these will provide a theoretical framework to guide future experiments into how complex cognitive functions originate and how they influence sensory perception and lead to robust behavioral performance.The planned projects will be organized into two flavors. The first attempts to borrow from existing neuromorphic approaches that rely on cortical representations to develop new embeddings within the deep neural networks framework, which will in turn endow the latter with brain-like robustness in challenging unanticipated environments. Three specific efforts within this flavor will be conducted: Learning DNN embeddings using cortical representations of speech and music, exploring unsupervised clustering of cortical features using adversarial auto-encoders, and exploiting pitch and timbre representations to enhance segregation of sound. The second flavor of projects borrows from the DNN approach to build into neuromorphic algorithms the desirable performance and flexibility attained by training on available databases. Two broad areas of studies are planned: one focuses on questions of neuromorphic implementations that benefit from DNN toolboxes and ideas, especially in segregation and reconstruction. The other focuses on investigating how autoencoders can be exploited to implement feature reduction and clustering efficiently.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目研究了如何适应和应用大脑皮层中语音和音乐的神经表征，以克服在极其嘈杂和混乱的环境中强大感知的挑战，模仿大脑的处理和能力。更具体地说，该项目将制定受大脑结构启发的算法，以隔离和跟踪目标扬声器或声源，测试其性能，并将其与利用深度人工神经网络完成这些任务的最先进方法相关联。人类心理声学和生理实验与这些算法将进行测试的有效性，这些想法模仿人类的能力。这一努力将刺激以大脑及其认知功能为模型的新神经形态计算工具的发展。反过来，这些将提供一个理论框架，指导未来的实验，以了解复杂的认知功能是如何产生的，以及它们如何影响感官知觉，并导致强大的行为表现。第一种尝试借鉴现有的神经形态方法，这些方法依赖于皮层表征来在深度神经网络框架内开发新的嵌入，这反过来又将赋予后者在具有挑战性的意外环境中具有类似大脑的鲁棒性。在这种风格中将进行三项具体的工作：使用语音和音乐的皮层表示来学习DNN嵌入，使用对抗性自动编码器探索皮层特征的无监督聚类，以及利用音高和音色表示来增强声音的分离。第二种风格的项目借鉴DNN方法，将通过在可用数据库上训练获得的理想性能和灵活性构建到神经形态算法中。计划进行两个广泛的研究领域：一个侧重于从DNN工具箱和想法中受益的神经形态实现问题，特别是在隔离和重建方面。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Harmonicity Plays a Critical Role in DNN Based Versus in Biologically-Inspired Monaural Speech Segregation Systems

和谐性在基于 DNN 的系统与受生物启发的单耳语音分离系统中发挥着关键作用

• DOI: 10.1109/icassp43922.2022.9747314
• 发表时间: 2022
• 期刊: Speech and Signal Processing
• 作者: Parikh, Rahil;Kavalerov, Ilya;Espy-Wilson, Carol;Shamma, Shihab
• 通讯作者: Shamma, Shihab

The Mirrornet : Learning Audio Synthesizer Controls Inspired by Sensorimotor Interaction

镜网：受感觉运动交互启发学习音频合成器控制

• DOI: 10.1109/icassp43922.2022.9747358
• 发表时间: 2022
• 期刊: Speech and Signal Processing
• 作者: Siriwardena, Yashish M.;Marion, Guilhem;Shamma, Shihab
• 通讯作者: Shamma, Shihab

Acoustic To Articulatory Speech Inversion Using Multi-Resolution Spectro-Temporal Representations Of Speech Signals

使用语音信号的多分辨率时谱表示的声学到发音语音反演

• 发表时间: 2022
• 期刊: Interspeech
• 作者: R Parikh, N Seneviratne

Unsupervised speaker adaptation for speaker independent acoustic to articulatory speech inversion

用于独立于说话人的声学到发音语音反转的无监督说话人自适应

• DOI: 10.1121/1.5116130
• 发表时间: 2019
• 期刊: The Journal of the Acoustical Society of America
• 作者: Sivaraman, Ganesh;Mitra, Vikramjit;Nam, Hosung;Tiede, Mark;Espy-Wilson, Carol
• 通讯作者: Espy-Wilson, Carol