Audio and Video Based Speech Separation for Multiple Moving Sources Within a Room Environment

针对房间环境内多个移动源的基于音频和视频的语音分离

• 批准号: EP/H049665/1
• 负责人: Jonathon Chambers
• 金额: $ 38.32万
• 依托单位: Loughborough University
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2010
• 资助国家: 英国
• 起止时间: 2010 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FH049665%2F1
• 关键词: Audio; Video; Based; Speech; Separation

Human beings have developed a unique ability to communicate within a noisy environment, such as at a cocktail party. This skill is dependent upon the use of both the aural and visual senses together with sophisticated processing within the brain. To mimic this ability within a machine is very challenging, particularly if the humans are moving, such as in a teleconferencing context, when human speakers are walking around a room. In the field of signal processing researchers have developed techniques to separate one speech signal from a mixture of such signals, as would be measured by a number of microphones, on the basis of only audio information with the assumption that the humans are static and typically no more than two humans are within the room. Such approaches have generally been found to fail, however, when the human speakers are moving and when there are more than two in number. Fundamentally new approaches are therefore necessary to advance the state-of-the-art in the field. Professor Chambers and his team at Loughborough University were the first in the UK to propose a new approach on the basis of combined audio and video processing to solve the source separation problem, but their preliminary approach identified major challenges in audio-visual speaker localization, tracking and separation which must be solved to provide a practical solution for speech separation for multiple moving sources within a room environment. These findings motivate this new project in which world-leading teams at the University of Surrey, led by Professor Kittler, and at the GIPSA Lab, Grenoble, France, headed by Professor Jutten, are ready to work with Professor Chambers and his team at Loughborough University to advance the state-of-the-art in the field.In this new project, two postdoctoral researchers will be employed, one at Loughborough and another at Surrey. The first will focus on the development of fundamentally new speech source separation algorithms for moving speakers by using geometrical room acoustic (for example location and number of sources, descriptions of their movement) information provided by the second researcher. The research team at Grenoble will provide technical guidance on the basis of their considerable experience in source separation throughout the project and will work on providing an acoustic noise model for the room environment which will also aid the speech separation process. To achieve these tasks, frequency domain based beamforming algorithms will be developed which exploit microphone arrays having more microphones than speakers so that new data independent superdirective robust beamformer design methods can be exploited using mathematical convex optimization. Additionally, further geometic information will be exploited to introduce robustness to errors in the localization information describing the desired source and the interference. To improve the localization information an array of collaborative cameras will be used and both audio and visual information will be used. Advanced methods from particle filtering and probabilistic data association will be exploited for improving the tracking performance. Finally, visual voice activity detection will be used to determine the active sources within the beamforming operations. We emphasize that this work is not implementation-driven, so computational complexity for real-time realization will not be a focus; this would be the subject of a future project.All of the new algorithms will be evaluated both in terms of objective and subjective performance measures on labelled audio and visual datasets acquired at Loughbourgh and Surrey, and from the CHIL seminar room at the Karlsruhe University (UKA), Germany. To ensure this pioneering work has maximum impact on the UK and international academic and research communities all the algorithms and datasets will be made available through the project website.

人类已经发展出一种独特的能力，可以在嘈杂的环境中进行交流，比如在鸡尾酒会上。这种技能依赖于听觉和视觉的使用以及大脑内部复杂的处理。在机器中模仿这种能力是非常具有挑战性的，特别是如果人类正在移动，例如在电话会议的背景下，当人类说话者在房间里走动时。在信号处理领域中，研究人员已经开发了仅基于音频信息从这样的信号的混合中分离一个语音信号的技术，其中假设人是静态的并且通常不超过两个人在房间内，所述语音信号将由多个麦克风测量。然而，当人类说话者正在移动并且数量超过两个时，这种方法通常被发现失败。因此，必须从根本上采取新的方法，以推进该领域的最新技术。拉夫堡大学的Chambers教授和他的团队是英国第一个提出一种基于组合音频和视频处理的新方法来解决源分离问题的人，但他们的初步方法确定了视听扬声器定位，跟踪和分离方面的主要挑战，必须解决这些挑战，才能为室内环境中多个移动源的语音分离提供实用的解决方案。这些发现激发了这个新项目，由Kittler教授领导的萨里大学和由Jutten教授领导的法国格勒诺布尔GIPSA实验室的世界领先团队准备与拉夫堡大学的Chambers教授和他的团队合作，推进该领域的最先进技术。在这个新项目中，将雇用两名博士后研究人员，一名在拉夫堡，另一名在萨里。第一个将专注于开发从根本上新的语音源分离算法，移动扬声器通过使用几何室声学（例如位置和数量的源，他们的运动的描述）的第二个研究人员提供的信息。格勒诺布尔的研究团队将根据他们在整个项目中源分离方面的丰富经验提供技术指导，并将致力于为室内环境提供声学噪声模型，这也将有助于语音分离过程。为了实现这些任务，将开发基于频域的波束形成算法，其利用具有比扬声器更多的麦克风的麦克风阵列，使得可以使用数学凸优化来利用新的数据独立的超定向鲁棒波束形成器设计方法。另外，将利用进一步的几何信息来引入对描述期望源和干扰的定位信息中的误差的鲁棒性。为了改进定位信息，将使用一系列协作相机，并将使用音频和视频信息。粒子滤波和概率数据关联的先进方法将被用来提高跟踪性能。最后，视觉语音活动检测将用于确定波束成形操作内的活动源。我们强调，这项工作是不是实现驱动的，所以计算的复杂性，实时实现将不会是一个焦点，这将是未来的一个项目的主题。所有的新算法将进行评估，无论是在客观和主观的性能指标上标记的音频和视觉数据集在拉夫堡和萨里，并从CHIL研讨室在卡尔斯鲁厄大学（UKA），德国。为了确保这项开创性工作对英国和国际学术和研究界产生最大影响，所有算法和数据集都将通过项目网站提供。

项目成果

Fast pose invariant face recognition using super coupled multiresolution Markov Random Fields on a GPU

在 GPU 上使用超耦合多分辨率马尔可夫随机场进行快速姿势不变人脸识别

• DOI: 10.1016/j.patrec.2014.05.017
• 发表时间: 2014
• 期刊: Pattern Recognition Letters
• 影响因子: 5.1
• 作者: Rahimzadeh Arashloo S

Multimodal blind source separation with a circular microphone array and robust beamforming

• 发表时间: 2011-08
• 期刊: 2011 19th European Signal Processing Conference
• 作者: S. M. Naqvi;Muhammad Salman Khan;Qingju Liu;Wenwu Wang;J. Chambers

Robust Feature Selection for Scaling Ambiguity Reduction in Audio-Visual Convolutive BSS

用于音视频卷积 BSS 中缩放模糊度减少的鲁棒特征选择

• 作者: Syed Mohsen Naqvi (Author)

A Multimodal Approach to Blind Source Separation of Moving Sources

移动源盲源分离的多模态方法

• DOI: 10.1109/jstsp.2010.2057198
• 发表时间: 2010
• 期刊: IEEE Journal of Selected Topics in Signal Processing
• 影响因子: 7.5
• 作者: Naqvi S

Audio video based fast fixed-point independent vector analysis for multisource separation in a room environment

• DOI: 10.1186/1687-6180-2012-183
• 发表时间: 2012-08
• 期刊: EURASIP Journal on Advances in Signal Processing
• 影响因子: 1.9
• 作者: Yanfeng Liang;S. M. Naqvi;J. Chambers