CISE-ANR: HCC: Small: Omnidirectional BatVision: Learning How to Navigate from Cell Phone Audios

该项目的目标是开发实时3D空间重建，该声音不是由昂贵的专门设备捕获的，而是由普通消费级手机捕获的。这种方法的灵感来自蝙蝠使用的回声定位，将从仅有声音的3D空间地图开发出足以用于导航的3D空间地图，例如近距离避障和在拥挤的火车站找到遥远的出口。这项研究将使3D视觉超越视线，在弱光或无光的条件下，应用范围从能听到拐角处行人的监听汽车到从人群中集体重建3D地图。项目成果将有助于更好地对机器人中的声音感知和有效的声音-视觉一体化进行计算建模，并有助于有效的应用，例如为视障人士和因烟雾或黑暗造成的低能见度条件下的消防员提供导航辅助设备。这项工作将为视觉3D地图提供一个互补的、成本效益高的替代方案，让每个人都能成为3D内容创作者。从声音中感知3D是一项具有挑战性的任务。虽然立体声音频为水平到达方向估计提供了直接提示，但它只在控制良好的环境中起作用。在现实世界中，没有简单的数学模型可以将声音映射到3D空间，因为设备方向、房间布局、材料、背景噪音等许多因素都会影响声音的传播。这个项目采用了一种机器学习的方法来从手机音频中推断3D空间。将使用带有双耳麦克风、扬声器和RGB-D立体声的传感器装置在不同环境中收集大规模视听数据集。连接的智能手机将使用自己的立体声麦克风和摄像头记录时间同步的数据。扬声器将发出信号以实现回声定位，但部分数据将仅包含自然发生的声音。几个带有激光雷达扫描的3D模型的室内和室外环境将作为地面真实情况。还将在公共街道收集数据，以测试在无法进行激光雷达扫描的现实情况下的健壮性。在给定数据集的情况下，将针对视场和360°全景制定几个3D场景重建任务，首先使用特权传感器数据，最后仅使用手机传感器。在使用双耳麦克风和立体声摄像机在各种环境中收集大规模视听数据后，将训练一个模型将声音数据映射到从视觉数据提取的深度图。一旦模型经过训练，它将能够仅根据声音输入来“看到”3D空间。然后，该模型将被改装，以实现与手机上提供的立体声麦克风和传感器相同的高质量3D感知。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to develop real-time 3D space reconstruction from sound captured not by expensive specialized equipment, but by common-place consumer-grade mobile phones. The approach, inspired by echolocation used by bats, is to develop from sound alone 3D spatial maps that are sufficient for navigation, such as close obstacle avoidance and finding distant exits in a crowded train station. The research will enable 3D vision beyond the line of sight and in low or no light conditions with applications ranging from listening cars that can hear pedestrians around the corner to collective 3D map reconstruction from crowds. Project outcomes will contribute to better computational modeling of sound perception and effective sound-vision integration in robotics, as well as to impactful applications such as navigational aids for visually impaired persons and for fire-fighters in low visibility conditions caused by smoke or darkness. The work will provide a complementary cost-effective alternative to visual 3D mapping that allows everybody to become a 3D content creator.The task of 3D perception from sound is challenging. While stereo audio provides direct cues for horizontal direction of arrival estimation, it only works in well controlled environments. There are no simple mathematical models to map sound to 3D space in real-word settings, as many factors such as device orientations, room layouts, materials, background noises shape sound propagation. This project takes a machine learning approach to infer 3D space from cell phone audios. A large-scale audio-visual dataset will be collected in different environments using a sensor-rig with a binaural microphone, a speaker and an RGB-D stereo. An attached smartphone will record time-synchronized data with its own stereo microphone and cameras. The speaker will emit signals to enable echolocation, but a part of the data will contain only naturally occurring sounds. Several indoor and outdoor environments with LiDAR scanned 3D models will serve as ground-truth. Data will also be collected in public streets to test robustness in realistic situations where LiDAR scans are not possible. Given the dataset, several 3D scene reconstruction tasks will be formulated for both the field of view and full 360° view, first with privileged sensor data and finally from cellphone sensors alone. After collecting large-scale audio-visual data in a variety of environments with binaural microphones and stereo cameras, a model will be trained to map sound data to depth maps extracted from visual data. Once the model is trained, it will be able to “see” the 3D space based on sound inputs alone. The model will then be adapted to achieve the same high quality 3D perception with stereo-microphones and sensors available on a mobile phone.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.