Collaborative Research:CIF:Small:Acoustic-Optic Vision - Combining Ultrasonic Sonars with Visible Sensors for Robust Machine Perception

计算机视觉主要使用光学信息来理解环境，从物体检测到机器人定位和绘图。然而，自然界中的许多动物使用多种传感方式来探索它们的环境。特别是，听觉和声学已经有效地与视觉一起用于增强野外导航。海豚和蝙蝠都使用声纳回声定位来帮助他们在水下或夜间等视觉退化的环境中定位。受其哺乳动物传感系统的启发，该项目研究了光学和超声波传感器的融合，以增强计算机视觉和计算成像。研究团队将开发新型硬件平台，用于执行联合光学和声学传感，以及相应的图形和视觉算法，以支持多模态数据的模拟和分析。由此产生的传感平台将用于3D扫描、材料识别、非视距成像和机器人测绘。此外，该项目通过将生物视觉和声学融入计算机图形学和视觉课程、本科研究、与产学研联盟成员的合作以及中学外展计划，实现了传播和教育目标。该项目将侧重于三个主要目标。第一个目标将是设计新的渲染算法，可以在一个环境中联合合成光学和声学信息。这些算法将通过在环境中对光线和声波进行重要采样而变得有效，并将告知传感器模型和采样标准以恢复环境信息。第二个目标将是开发新的传感模型，以权衡跨光学和声学模式的采样。这些系统将利用多个图像传感器和声学换能器，这些传感器将在实际硬件中原型化。最终的目标将集中在机器人系统的视觉和声学相结合的共同利益。将研究在3D传感、材料识别、非视距/闭塞成像和声光机器人导航方面的应用。研究见解将通过科学出版物和演示以及开源软件和硬件设计的可重复性传播到机器人，计算机视觉，声学和计算成像社区。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Computer vision has primarily used optical information to understand the environment, from object detection to robotic localization and mapping. However, many animals in the natural world use multiple sensing modalities to explore their environments. In particular, hearing and acoustics have been used effectively with vision to enhance navigation in the wild. Dolphins and bats both use sonar echolocation to help triangulate their position in visually degraded environments such as underwater or at night. Inspired by their mammalian sensing systems, this project investigates the fusion of optical and ultrasonic sensors for enhanced computer vision and computational imaging. The team of researchers will develop novel hardware platforms for performing joint optical and acoustic sensing along with the corresponding graphics and vision algorithms to support simulation and analysis of this multimodal data. The resulting sensing platforms will be leveraged for applications in 3D scanning, material recognition, non-line-of-sight imaging, and robotic mapping. Further, the project features dissemination and education goals through incorporation of bio-inspired vision and acoustics into computer graphics and vision courses, undergraduate research, engagement with members of an industry-university consortium, and a middle-school outreach program.The project will focus on three main objectives. The first objective will be to design new rendering algorithms that can jointly synthesize optical and acoustical information in an environment. These algorithms will be made efficient by the importance sampling of optical rays and sound waves in an environment and will inform sensor models and sampling criteria to recover the environment information. The second objective will be to develop new sensing models that trade off sampling across optical and acoustic modalities. These systems will leverage multiple image sensors and acoustic transducers which will be prototyped in real hardware. The final objective will focus on the mutual benefit of combining vision and acoustics for robotics systems. Applications in 3D sensing, material identification, non-line-of-sight/occluded imaging, and acoustic-optic robotic navigation will be investigated. Research insights will be disseminated to the robotics, computer-vision, acoustics, and computational-imaging communities through scientific publications and presentations as well as open-source software and hardware designs for reproducibility.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.