权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Deep Learning Based Complex Spectral Mapping for Multi-Channel Speaker Separation and Speech Enhancement

基于深度学习的复杂频谱映射，用于多通道说话人分离和语音增强

基本信息

批准号：
2125074
负责人：
Eric Fosler-Lussier
金额：
$ 39.06万
依托单位：
Ohio State University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2021
资助国家：
美国
起止时间：
2021-08-01 至 2024-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2125074&HistoricalAwards=false
关键词：
Deep Learning Based Complex Spectral

项目摘要

Despite tremendous advances in deep learning based speech separation and automatic speech recognition, a major challenge remains how to separate concurrent speakers and recognize their speech in the presence of room reverberation and background noise. This project will develop a multi-channel complex spectral mapping approach to multi-talker speaker separation and speech enhancement in order to improve speech recognition performance in such conditions. The proposed approach trains deep neural networks to predict the real and imaginary parts of individual talkers from the multi-channel input in the complex domain. After overlapped speakers are separated into simultaneous streams, sequential grouping will be performed for speaker diarization, which is the task of grouping the speech utterances of the same talker over intervals with the utterances of other speakers and pauses. Proposed speaker diarization will integrate spatial and spectral speaker features, which will be extracted through multi-channel speaker localization and single-channel speaker embedding. Recurrent neural networks will be trained to perform classification for the purpose of speaker diarization, which can handle an arbitrary number of speakers in a meeting. The proposed separation system will be evaluated using open, multi-channel speaker separation datasets that contain both room reverberation and background noise. The results from this project are expected to substantially elevate the performance of continuous speaker separation, as well as speaker diarization, in adverse acoustic environments, helping to close the performance gap between recognizing single-talker speech and recognizing multi-talker speech.The overall goal of this project is to develop a deep learning system that can continuously separate individual speakers in a conversational or meeting setting and accurately recognize the utterances of these speakers. Building on recent advances on simultaneous grouping to separate and enhance overlapped speakers in a talker-independent fashion, the project is mainly focused on speaker diarization, which aims to group the speech utterances of the same speaker across time. To achieve speaker diarization, deep learning based sequential grouping will be performed and it will integrate spatial and spectral speaker characteristics. Through sequential organization, simultaneous streams will be grouped with earlier-separated speaker streams to form sequential streams, each of which corresponds to all the utterances of the same speaker up to the current time. Speaker localization and classification will be investigated to make sequential grouping capable of creating new sequential streams and handling an arbitrary number of speakers in a meeting scenario. With the added spatial dimension, the proposed diarization approach provides a solution to the question of who spoke when and where, significantly expanding the traditional scope of who spoke when. The proposed separation system will be evaluated using multi-channel speaker separation datasets that contain highly overlapped speech in recorded conversations, as well as room reverberation and background noise present in real environments. The main evaluation metric will be word error rate in automatic speech recognition. The performance of speaker diarization will be measured using diarization error rate.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管基于深度学习的语音分离和自动语音识别取得了巨大进步，但主要挑战仍然是如何在存在室内混响和背景噪声的情况下分离并发说话者并识别他们的语音。该项目将开发一种多通道复杂谱映射方法来进行多说话者说话者分离和语音增强，以提高此类条件下的语音识别性能。所提出的方法训练深度神经网络，以从复域中的多通道输入预测个体说话者的真实的和虚部。在重叠的说话者被分离成同时的流之后，将执行用于说话者日志化的顺序分组，其是将同一说话者的语音话语在间隔上与其他说话者的话语和停顿分组的任务。提出的说话人日志化将空间和频谱说话人特征相结合，通过多通道说话人定位和单通道说话人嵌入来提取。循环神经网络将被训练来执行分类，以用于扬声器日记化，它可以处理会议中任意数量的扬声器。建议的分离系统将使用开放的，多通道扬声器分离数据集，包含房间混响和背景噪声进行评估。该项目的结果预计将大大提高连续扬声器分离的性能，以及扬声器日记，在恶劣的声学环境，有助于缩小识别单个说话者语音和识别多个说话者语音之间的性能差距，该项目的总体目标是开发一个深度学习系统，该系统可以在对话或会议环境中连续分离单个发言者，并准确地识别每个发言者。识别这些说话者的话语。该项目基于最近同步分组的进展，以独立于说话者的方式分离和增强重叠的说话者，主要集中在说话者日记化上，其目的是将同一说话者的语音话语在不同时间分组。为了实现说话人日志化，将执行基于深度学习的顺序分组，并将整合空间和频谱说话人特征。通过顺序组织，同时流将与较早分离的说话者流分组以形成顺序流，每个顺序流对应于同一说话者直到当前时间的所有话语。扬声器定位和分类将进行调查，使顺序分组能够创建新的顺序流和处理会议场景中的任意数量的扬声器。随着空间维度的增加，所提出的日记化方法为谁在何时何地发言的问题提供了一个解决方案，大大扩展了谁在何时发言的传统范围。所提出的分离系统将使用多声道扬声器分离数据集进行评估，这些数据集包含记录的对话中的高度重叠的语音，以及存在于真实的环境中的房间混响和背景噪声。在自动语音识别中，主要的评价指标是词的错误率。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。