Robot Audition in Everyday Environments with Deep Learning

人类和动物依靠听觉来监测环境，探测威胁和交流。尤其是听觉作为瞬时的全向注意机制，而视觉则具有更多的连续定向视野。机器人应该具有类似的听觉能力，以便与环境自然互动。处理嘈杂和动态的环境是一项具有挑战性的任务，即在我们生活的狂野和混乱的世界中，机器人必须实时执行所有计算。机器人试镜将使机器将注意力集中在视觉无法感知的高优先级事件上。在没有其他接口可用的情况下，人类也可以向机器人发出语音命令。我相信，如果智能机器拥有先进的听力能力，那么在日常环境中大规模部署机器人是可能的。这项发现基金的目标是研究新的方法，使机器人在现实世界中具有类似或优于人类的听力能力。深度学习在音频处理方面显示出了很好的结果，但它依赖于大型数据集，并且在测试时需要很高的计算能力。深度学习方法需要适应机器人，因为嵌入式计算能力和训练数据量通常都是有限的。为了实现这一目标，基于我目前的研究背景，我计划解决以下四个短期目标：1)开发一种迁移学习方法来执行少镜头自我噪声学习；2)基于大型弱标记数据集进行声音事件定位；3)利用神经网络对任意形状阵列的每对麦克风进行时频掩模估计，增强语音源；4)对已有的神经网络进行优化，使其能够在嵌入式系统上进行实时处理。这些方法将在与现实环境相互作用的轮式机器人上进行验证。为了最大限度地影响机器人社区，所有的方法都将集成在一个开源软件框架中。该项目将培养4名高素质人才（2名博士和2名硕士）和4名本科合作培训生。由HPQs开发的专业知识将基于信号处理，机器学习和嵌入式系统设计的独特融合，这在加拿大机器人领域的公司中需求量很大。机器人被认为是解决发达国家人口老龄化和劳动力短缺问题的一部分。机器人试听将允许机器与仓库和智能工厂的工人合作，并使机器人技术对老年人和医疗保健更安全。由于智能音箱的技术依赖于云计算，因此在隐私方面也存在一些严重的担忧。我的研究项目超越了机器人技术，可以使这个行业受益，因为目前研究的一部分旨在在低成本嵌入式硬件上进行语音增强。

Humans and animals rely on audition to monitor their environment, to detect threats and to communicate. In particular, hearing acts as instantaneous omnidirectional attention mechanism, while vision has more continuous and directed field of view. Robots should have similar hearing capabilities to naturally interact with their environment. It is a challenging task to address noisy and dynamic environments, i.e. in the wild and messy world we live in, with the robot having to perform all computations in real-time. Robots audition will allow machines to focus their attention on high priority events not perceptible by vision. Humans could also give robots voice commands when no other interface is available. I believe that large scale deployment of robots in everyday environments will be possible if intelligent machines have advanced hearing capabilities. The goal of this Discovery Grant is to investigate new methods to provide robots with hearing capabilities similar or superior to humans in real world setting. Deep learning showed promising results in audio processing but relies on large datasets and involves high computing power at test time. Deep learning methods need to be adapted for robots as both the embedded computing power and the amount of training data are usually limited. To accomplish this, and based on my current research background, I plan to address the following four short-term objectives: 1) develop a transfer learning method to perform few-shot ego-noise learning; 2) perform sound event localization based on large weakly-labeled datasets; 3) enhance a speech source with neural networks estimating time-frequency masks for each pair of microphones for an array of arbitrary shape; 4) optimize the previous neural networks for real-time processing on an embedded system. The methods will be validated on a wheeled robot that interacts with its environment in realistic settings. To maximize the impact on the robotics community, all the methods will be integrated in an open source software framework. This program will train 4 Highly Qualified Personnel (HQPs) (2 PhDs and 2 MScAs) and 4 undergrad coop trainees. The expertise developed by the HPQs will be based on a unique blend of signal processing, machine learning and embedded system design, which is in high demand amongst Canadian companies involved in the field of robotics. Robots are believed to be part of the solutions to deal with the aging population and labor shortage in developed countries. Robot audition would allow machine to collaborate with workers in warehouses and smart plants and make robotic technologies safer for seniors and healthcare. There is also some serious concern regarding privacy with smart speakers as the technology relies on cloud computing. My research program goes beyond robotics and can benefit this industry as some part of the current research aims to perform speech enhancement on low-cost embedded hardware.