Robot Audition in Everyday Environments with Deep Learning

点击翻译按钮获取中文摘要

Humans and animals rely on audition to monitor their environment, to detect threats and to communicate. In particular, hearing acts as instantaneous omnidirectional attention mechanism, while vision has more continuous and directed field of view. Robots should have similar hearing capabilities to naturally interact with their environment. It is a challenging task to address noisy and dynamic environments, i.e. in the wild and messy world we live in, with the robot having to perform all computations in real-time. Robots audition will allow machines to focus their attention on high priority events not perceptible by vision. Humans could also give robots voice commands when no other interface is available. I believe that large scale deployment of robots in everyday environments will be possible if intelligent machines have advanced hearing capabilities. The goal of this Discovery Grant is to investigate new methods to provide robots with hearing capabilities similar or superior to humans in real world setting. Deep learning showed promising results in audio processing but relies on large datasets and involves high computing power at test time. Deep learning methods need to be adapted for robots as both the embedded computing power and the amount of training data are usually limited. To accomplish this, and based on my current research background, I plan to address the following four short-term objectives: 1) develop a transfer learning method to perform few-shot ego-noise learning; 2) perform sound event localization based on large weakly-labeled datasets; 3) enhance a speech source with neural networks estimating time-frequency masks for each pair of microphones for an array of arbitrary shape; 4) optimize the previous neural networks for real-time processing on an embedded system. The methods will be validated on a wheeled robot that interacts with its environment in realistic settings. To maximize the impact on the robotics community, all the methods will be integrated in an open source software framework. This program will train 4 Highly Qualified Personnel (HQPs) (2 PhDs and 2 MScAs) and 4 undergrad coop trainees. The expertise developed by the HPQs will be based on a unique blend of signal processing, machine learning and embedded system design, which is in high demand amongst Canadian companies involved in the field of robotics. Robots are believed to be part of the solutions to deal with the aging population and labor shortage in developed countries. Robot audition would allow machine to collaborate with workers in warehouses and smart plants and make robotic technologies safer for seniors and healthcare. There is also some serious concern regarding privacy with smart speakers as the technology relies on cloud computing. My research program goes beyond robotics and can benefit this industry as some part of the current research aims to perform speech enhancement on low-cost embedded hardware.