Robot Audition in Everyday Environments with Deep Learning

人类和动物依靠听觉来监控他们的环境，检测威胁和交流。特别是，听觉是一种瞬时的全方位注意机制，而视觉则具有更连续、更定向的视野。机器人应该具有类似的听力能力，才能与环境自然互动。这是一项具有挑战性的任务，以解决噪声和动态环境，即在我们生活的狂野和混乱的世界中，机器人必须实时执行所有计算。机器人试镜将允许机器将注意力集中在视觉无法感知的高优先级事件上。当没有其他接口可用时，人类还可以向机器人发出语音命令。我相信，如果智能机器具有先进的听力能力，机器人在日常环境中的大规模部署将是可能的。这项发现基金的目标是研究新的方法，为机器人提供与现实世界中的人类相似或更好的听力能力。深度学习在音频处理方面显示出了令人振奋的结果，但它依赖于大数据集，并且在测试时需要很高的计算能力。由于机器人的嵌入式计算能力和训练数据量通常都是有限的，因此需要采用深度学习方法。为了实现这一目标，并基于我目前的研究背景，我计划解决以下四个短期目标：1)开发一种转移学习方法来执行少镜头自噪声学习；2)基于大型弱标记数据集进行声音事件定位；3)使用神经网络估计任意形状阵列中每对麦克风的时频掩模来增强语音源；4)优化先前的神经网络，以便在嵌入式系统上进行实时处理。这些方法将在一个轮式机器人上进行验证，该机器人在现实环境中与环境互动。为了最大限度地扩大对机器人社区的影响，所有方法都将集成在一个开放源代码软件框架中。该计划将培训4名高素质人才(HQP)(2名博士和2名硕士)和4名本科生。HPQ开发的专业知识将基于信号处理、机器学习和嵌入式系统设计的独特组合，这在涉及机器人领域的加拿大公司中需求很高。机器人被认为是应对发达国家人口老龄化和劳动力短缺的解决方案的一部分。机器人试镜将允许机器与仓库和智能工厂中的工人合作，并使机器人技术对老年人和医疗保健更安全。还有一些关于智能扬声器隐私的严重担忧，因为这项技术依赖于云计算。我的研究项目超越了机器人学，可以让这个行业受益，因为当前研究的一部分旨在对低成本的嵌入式硬件进行语音增强。

Humans and animals rely on audition to monitor their environment, to detect threats and to communicate. In particular, hearing acts as instantaneous omnidirectional attention mechanism, while vision has more continuous and directed field of view. Robots should have similar hearing capabilities to naturally interact with their environment. It is a challenging task to address noisy and dynamic environments, i.e. in the wild and messy world we live in, with the robot having to perform all computations in real-time. Robots audition will allow machines to focus their attention on high priority events not perceptible by vision. Humans could also give robots voice commands when no other interface is available. I believe that large scale deployment of robots in everyday environments will be possible if intelligent machines have advanced hearing capabilities. The goal of this Discovery Grant is to investigate new methods to provide robots with hearing capabilities similar or superior to humans in real world setting. Deep learning showed promising results in audio processing but relies on large datasets and involves high computing power at test time. Deep learning methods need to be adapted for robots as both the embedded computing power and the amount of training data are usually limited. To accomplish this, and based on my current research background, I plan to address the following four short-term objectives: 1) develop a transfer learning method to perform few-shot ego-noise learning; 2) perform sound event localization based on large weakly-labeled datasets; 3) enhance a speech source with neural networks estimating time-frequency masks for each pair of microphones for an array of arbitrary shape; 4) optimize the previous neural networks for real-time processing on an embedded system. The methods will be validated on a wheeled robot that interacts with its environment in realistic settings. To maximize the impact on the robotics community, all the methods will be integrated in an open source software framework. This program will train 4 Highly Qualified Personnel (HQPs) (2 PhDs and 2 MScAs) and 4 undergrad coop trainees. The expertise developed by the HPQs will be based on a unique blend of signal processing, machine learning and embedded system design, which is in high demand amongst Canadian companies involved in the field of robotics. Robots are believed to be part of the solutions to deal with the aging population and labor shortage in developed countries. Robot audition would allow machine to collaborate with workers in warehouses and smart plants and make robotic technologies safer for seniors and healthcare. There is also some serious concern regarding privacy with smart speakers as the technology relies on cloud computing. My research program goes beyond robotics and can benefit this industry as some part of the current research aims to perform speech enhancement on low-cost embedded hardware.