Audio and Video Based Speech Separation for Multiple Moving Sources Within a Room Environment

人类已经发展出一种独特的能力，可以在嘈杂的环境中进行交流，比如在鸡尾酒会上。这种技能依赖于听觉和视觉感官的使用以及大脑内部复杂的处理。在机器中模仿这种能力是非常具有挑战性的，特别是当人类在移动时，比如在电话会议环境中，当人类在房间里走来走去时。在信号处理领域，研究人员已经开发出一种技术，可以将一个语音信号从这些信号的混合物中分离出来，这些信号将由许多麦克风测量，仅基于音频信息，假设人类是静态的，通常不超过两个人在房间里。然而，当说话人在移动时，当说话人的数量超过两个时，这种方法通常会失败。因此，从根本上说，需要新的方法来推进该领域的最新技术。在英国，拉夫堡大学的钱伯斯教授和他的团队首先提出了一种基于结合音频和视频处理的新方法来解决源分离问题，但他们的初步方法确定了视听扬声器定位、跟踪和分离方面的主要挑战，必须解决这些挑战，才能为房间环境中多个移动源的语音分离提供实用的解决方案。这些发现激发了这个新项目，由Kittler教授领导的萨里大学和由Jutten教授领导的法国格勒诺布尔GIPSA实验室的世界领先团队准备与拉夫堡大学的Chambers教授及其团队合作，推进该领域的最新技术。在这个新项目中，将雇用两名博士后研究人员，一名在拉夫堡，另一名在萨里。第一个研究将集中于利用第二个研究人员提供的几何房间声学（例如源的位置和数量，它们的运动描述）信息，为移动的说话者开发全新的语音源分离算法。格勒诺布尔的研究团队将根据他们在整个项目中丰富的源分离经验提供技术指导，并将致力于为房间环境提供声学噪声模型，这也将有助于语音分离过程。为了实现这些任务，将开发基于频域的波束形成算法，该算法利用麦克风多于扬声器的麦克风阵列，以便利用数学凸优化开发新的数据无关的超指令鲁棒波束形成设计方法。此外，进一步的几何信息将被利用来引入鲁棒性，以消除描述所需源和干扰的定位信息中的误差。为了改进定位信息，将使用一组协作摄像机，并同时使用音频和视觉信息。将利用粒子滤波和概率数据关联等先进方法来提高跟踪性能。最后，将使用视觉语音活动检测来确定波束形成操作中的有源。我们强调这项工作不是实现驱动的，因此实时实现的计算复杂性不会成为焦点；这将是未来项目的主题。所有的新算法都将根据在拉夫堡和萨里获得的标记音频和视觉数据集以及德国卡尔斯鲁厄大学（UKA）的CHIL研讨室获得的客观和主观性能指标进行评估。为了确保这项开创性的工作对英国和国际学术和研究界产生最大的影响，所有的算法和数据集都将通过项目网站提供。

Human beings have developed a unique ability to communicate within a noisy environment, such as at a cocktail party. This skill is dependent upon the use of both the aural and visual senses together with sophisticated processing within the brain. To mimic this ability within a machine is very challenging, particularly if the humans are moving, such as in a teleconferencing context, when human speakers are walking around a room. In the field of signal processing researchers have developed techniques to separate one speech signal from a mixture of such signals, as would be measured by a number of microphones, on the basis of only audio information with the assumption that the humans are static and typically no more than two humans are within the room. Such approaches have generally been found to fail, however, when the human speakers are moving and when there are more than two in number. Fundamentally new approaches are therefore necessary to advance the state-of-the-art in the field. Professor Chambers and his team at Loughborough University were the first in the UK to propose a new approach on the basis of combined audio and video processing to solve the source separation problem, but their preliminary approach identified major challenges in audio-visual speaker localization, tracking and separation which must be solved to provide a practical solution for speech separation for multiple moving sources within a room environment. These findings motivate this new project in which world-leading teams at the University of Surrey, led by Professor Kittler, and at the GIPSA Lab, Grenoble, France, headed by Professor Jutten, are ready to work with Professor Chambers and his team at Loughborough University to advance the state-of-the-art in the field.In this new project, two postdoctoral researchers will be employed, one at Loughborough and another at Surrey. The first will focus on the development of fundamentally new speech source separation algorithms for moving speakers by using geometrical room acoustic (for example location and number of sources, descriptions of their movement) information provided by the second researcher. The research team at Grenoble will provide technical guidance on the basis of their considerable experience in source separation throughout the project and will work on providing an acoustic noise model for the room environment which will also aid the speech separation process. To achieve these tasks, frequency domain based beamforming algorithms will be developed which exploit microphone arrays having more microphones than speakers so that new data independent superdirective robust beamformer design methods can be exploited using mathematical convex optimization. Additionally, further geometic information will be exploited to introduce robustness to errors in the localization information describing the desired source and the interference. To improve the localization information an array of collaborative cameras will be used and both audio and visual information will be used. Advanced methods from particle filtering and probabilistic data association will be exploited for improving the tracking performance. Finally, visual voice activity detection will be used to determine the active sources within the beamforming operations. We emphasize that this work is not implementation-driven, so computational complexity for real-time realization will not be a focus; this would be the subject of a future project.All of the new algorithms will be evaluated both in terms of objective and subjective performance measures on labelled audio and visual datasets acquired at Loughbourgh and Surrey, and from the CHIL seminar room at the Karlsruhe University (UKA), Germany. To ensure this pioneering work has maximum impact on the UK and international academic and research communities all the algorithms and datasets will be made available through the project website.