SFM++: Toward Active, Collaborative, and Semantic Structure-from-Motion (SFM)

点击翻译按钮获取中文摘要

Structure-from-Motion (SFM) techniques recover 3D information from 2D images. SFM has many applications, including 3D sensing in robotics and 3D content creation in computer games or movies. This proposal studies SFM from a unique perspective. I will take input from multiple independently moving video cameras with wide baselines to address 3D vision challenges. In comparison, previous methods often employ either a single camera or an array of fixed cameras with short baselines. These moving cameras can be handheld camcorders or wireless cameras mounted on different robot platforms. SFM in this novel setting is largely unexplored except in a few of our recent works [3, 4]. As demonstrated in these two works, this new SFM technique can be potentially used to guide a robot team for vision-based autonomous navigation or to provide a flexible handheld camcorder-based 3D reconstruction solution. This study has the potential to make a profound impact. It is the first attempt to develop a collaborative 3D vision system for a team of miniature robots. Robot vision is possibly the most important application of computer vision. Vision powered autonomous robots will bring revolutionary changes to society and benefit everybody. Due to the difficulties in computer vision, existing robot platforms often rely on bulky laser or ultrasonic scanners to solve the 3D sensing problem. However, these sensors are too heavy or energy inefficient for a miniature platform, e.g. a miniature robot with a payload of less than 50g. Vision is the ideal solution for 3D sensing on these miniature robots. Thus, the robotics and computer vision community devote tremendous efforts to designing 3D vision systems for robots. Almost all of these works focus on a single robot. To tackle difficult 3D vision problems, this proposal studies the collaborative vision sensing of a robot team. By exploiting the collaboration across multiple cameras, my students and I aim to develop robust and efficient computational algorithms for various 3D vision tasks. Specifically, we will address the robot team exploration problem to actively plan their navigation path to enhance 3D sensing. This includes maintaining view overlap between different robots and reducing measurement uncertainty. We will further develop real-time dense 3D reconstruction algorithms from multiple moving cameras, which can recover per-pixel depth information even under complicated dynamic environments. It effectively turns cameras into 3D scanners. We further plan to estimate skeletal human motion from the recovered 3D data by the collaboration of multiple cameras. This proposal will enable a team of miniature robots to navigate autonomously in complicated environments with moving objects. The robot team can sense skeletal human motion which enables natural human robot interaction. We will integrate and demonstrate our results with existing toy or commercial miniature unmanned aerial vehicles (UAVs) such as AR.Drone and Pelican. These algorithms can also be used for applications in computer graphics, such as flexible motion capture with handheld camcorders. This program is globally unique. The 3D vision of a collaborative robot team is a largely unexplored field. It will clearly extend Canada's lead in this area. Its results can be used in robotics applications for security/surveillance. The research result on 3D skeletal capture from handheld videos can be applied for flexible motion capture in the computer gaming industry, a major employer in Canada. Three PhD students will be trained in this project.