Collaborative Research:CIF:Small: Acoustic-Optic Vision - Combining Ultrasonic Sonars with Visible Sensors for Robust Machine Perception

计算机视觉主要使用光学信息来理解环境，从物体检测到机器人定位和映射。然而，自然世界中的许多动物使用多种传感方式来探索它们的环境。特别是，听觉和声学已经有效地与视觉一起使用，以增强野外导航。海豚和蝙蝠都使用声纳回声定位，以帮助三角测量他们的位置在视觉退化的环境，如水下或在夜间。受哺乳动物传感系统的启发，该项目研究了光学和超声波传感器的融合，以增强计算机视觉和计算成像。研究团队将开发新型硬件平台，用于执行联合光学和声学传感沿着相应的图形和视觉算法，以支持对这种多模态数据的模拟和分析。由此产生的传感平台将用于3D扫描、材料识别、非视线成像和机器人测绘等应用。此外，该项目还通过将生物启发的视觉和声学纳入计算机图形和视觉课程、本科生研究、与产学联盟成员的合作以及中学外展计划，实现传播和教育目标。该项目将重点关注三个主要目标。第一个目标将是设计新的渲染算法，可以在环境中联合合成光学和声学信息。这些算法将通过对环境中的光线和声波的重要性采样而变得高效，并且将通知传感器模型和采样标准以恢复环境信息。第二个目标是开发新的传感模型，在光学和声学模式之间权衡采样。这些系统将利用多个图像传感器和声学换能器，这些传感器将在真实的硬件中原型化。最终目标将集中在机器人系统视觉和声学相结合的互惠互利。将研究在3D传感，材料识别，非视线/遮挡成像和声光机器人导航中的应用。研究见解将通过科学出版物和演示以及开源软件和硬件设计的可重复性传播到机器人，计算机视觉，声学和计算成像社区。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Computer vision has primarily used optical information to understand the environment, from object detection to robotic localization and mapping. However, many animals in the natural world use multiple sensing modalities to explore their environments. In particular, hearing and acoustics have been used effectively with vision to enhance navigation in the wild. Dolphins and bats both use sonar echolocation to help triangulate their position in visually degraded environments such as underwater or at night. Inspired by their mammalian sensing systems, this project investigates the fusion of optical and ultrasonic sensors for enhanced computer vision and computational imaging. The team of researchers will develop novel hardware platforms for performing joint optical and acoustic sensing along with the corresponding graphics and vision algorithms to support simulation and analysis of this multimodal data. The resulting sensing platforms will be leveraged for applications in 3D scanning, material recognition, non-line-of-sight imaging, and robotic mapping. Further, the project features dissemination and education goals through incorporation of bio-inspired vision and acoustics into computer graphics and vision courses, undergraduate research, engagement with members of an industry-university consortium, and a middle-school outreach program.The project will focus on three main objectives. The first objective will be to design new rendering algorithms that can jointly synthesize optical and acoustical information in an environment. These algorithms will be made efficient by the importance sampling of optical rays and sound waves in an environment and will inform sensor models and sampling criteria to recover the environment information. The second objective will be to develop new sensing models that trade off sampling across optical and acoustic modalities. These systems will leverage multiple image sensors and acoustic transducers which will be prototyped in real hardware. The final objective will focus on the mutual benefit of combining vision and acoustics for robotics systems. Applications in 3D sensing, material identification, non-line-of-sight/occluded imaging, and acoustic-optic robotic navigation will be investigated. Research insights will be disseminated to the robotics, computer-vision, acoustics, and computational-imaging communities through scientific publications and presentations as well as open-source software and hardware designs for reproducibility.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.